[09:58:15] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367#11391188 (10ayounsi) One liners: `lang=python >>> spicerack.redfish('sretest2004').scp_dump().components['NIC.Integrated.1-1-1'].get('Broadcom_LLDPNearestBridg... [10:03:09] 10netops, 06Infrastructure-Foundations: rancid: message has lines too long for transport - https://phabricator.wikimedia.org/T410606 (10taavi) 03NEW [10:04:36] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391233 (10fgiunchedi) >>! In T407140#11383672, @cmooney wrote: > Ok thanks @fgiunchedi for the info. > > I think that seems doab... [10:07:15] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391235 (10taavi) >>! In T407140#11391233, @fgiunchedi wrote: > There are also NFS read only shares towards clouddump hosts, thoug... [12:21:37] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391608 (10cmooney) >>! In T407140#11391233, @fgiunchedi wrote: > The NFS shares host tools data and scratch space, specifically:... [12:32:04] 06Traffic, 10Hiddenparma, 13Patch-For-Review: Add ipblock-source objects and logic - https://phabricator.wikimedia.org/T402014#11391698 (10ops-monitoring-bot) Deployed hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - jayme@cumin2002 - T402014 [12:32:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11391704 (10cmooney) >>! In T408892#11389081, @Papaul wrote: > I think a am wrong on the public vlan for rack 22. We will not be re-imaging the servers in that... [12:51:26] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11391814 (10cmooney) [13:18:45] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11391938 (10ayounsi) Not sure if it has been discussed but what do you think of using Calico's [[ https://docs.tigera.io/calico/lat... [13:33:09] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11392000 (10cmooney) >>! In T407140#11391938, @ayounsi wrote: > But instead of doing network separation using a complex VRF and `ip... [13:47:22] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367#11392028 (10ayounsi) @papaul, could you have a look at the BIOS of sretest1005 ? The matching Redfish keys don't exist :( `lang=python >>> dump3.components['N... [13:51:43] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11392041 (10cmooney) [14:06:00] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11392104 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [14:31:56] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11392187 (10cmooney) [14:55:36] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11392286 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [15:19:01] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178#11392432 (10cmooney) [15:23:28] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11392495 (10MatthewVernon) [15:34:40] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11392629 (10MatthewVernon) [15:36:23] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: lvs1019: move primary uplink from asw2-c7-eqiad to lsw1-c7-eqiad and remove link to asw2-d2-eqiad - https://phabricator.wikimedia.org/T405628#11392634 (10cmooney) [15:55:53] vgutierrez: lvs1018 is in rack B7 connected to the switch there [15:56:02] but it also has links to row C and D switches: [15:56:08] https://netbox.wikimedia.org/dcim/interfaces/29150/trace/ [15:56:13] https://netbox.wikimedia.org/dcim/interfaces/29151/trace/ [15:56:42] ^^ do we need to move these terminations in row's C and D from the old Juniper to new Nokia switches as part of the switch migration? [15:56:59] or are we happy enough now with IPIP that we maybe should think about decom'ing those links instead? [15:57:23] I'd say we are happy enough with IPIP but I'll let sukhe answer given I'm OoO [15:57:34] oh sorry to bug you! [15:57:45] yeah I'll ask Sukhbir thanks [15:59:13] topranks: v.g said he is happy and I think that's the expectation we are going with in general, so +1 [16:01:44] sukhe: cool, thanks [16:01:55] yeah it's actually less work to move the link but we should probably just remove it at this point instead [16:16:07] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: lvs1018: decom links to asw2-c2-eqiad and asw2-d7-eqiad - https://phabricator.wikimedia.org/T410661 (10cmooney) 03NEW p:05Triage→03Medium [16:16:16] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: lvs1018: decom links to asw2-c2-eqiad and asw2-d7-eqiad - https://phabricator.wikimedia.org/T410661#11392925 (10cmooney) [16:16:20] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Eqiad row C/D switch refresh: LVS changes to support migration - https://phabricator.wikimedia.org/T405602#11392926 (10cmooney) [16:18:35] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11392932 (10MatthewVernon) [16:22:45] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367#11392975 (10Papaul) @ayounsi sretest1005 is the same as 2004 see below. what you can maybe check is the redfish /IDRAC version on sretest2004 and 1005 {F703... [16:28:45] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Compile a list of "canonical" thumbnail sizes - https://phabricator.wikimedia.org/T408715#11393052 (10MatthewVernon) [16:35:03] 06Traffic, 06Commons: Error: 503, Backend fetch failed - https://phabricator.wikimedia.org/T410201#11393109 (10RoyZuo) https://www.mediawiki.org/wiki/Phabricator/Help#Uploading_file_attachments The size of the uploaded files is limited to ~4 MB. I am unable to upload the file here. [16:54:00] 06Traffic, 06Commons: Error: 503, Backend fetch failed - https://phabricator.wikimedia.org/T410201#11393355 (10RoyZuo) i tried to paste the svg code using https://commons.wikimedia.org/wiki/User:Rillke/SVGedit.js into https://commons.wikimedia.org/wiki/File:Test.svg after a while a popup which says > Server... [16:56:31] 06Traffic, 06Commons: Error: 503, Backend fetch failed - https://phabricator.wikimedia.org/T410201#11393363 (10RoyZuo) i also tried https://commons.wikimedia.org/wiki/User:Rillke/bigChunkedUpload.js 00000: 1/10> in progress Upload: 100% 00004: 1/10> Chunk uploaded 00004: 2/10> in progress Upload: 100% 00007:... [16:58:24] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Servers exposing incorrect LLDP info - https://phabricator.wikimedia.org/T250367#11393374 (10ayounsi) Thanks, yeah that must be the reason : ` >>> spicerack.redfish('sretest1005').hw_model 9 >>> spicerack.redfish('sretest2004').hw_model 9 >... [17:11:56] 06Traffic, 06Commons: Error: 503, Backend fetch failed - https://phabricator.wikimedia.org/T410201#11393435 (10RoyZuo) there're bigger osm svg on commons like https://commons.wikimedia.org/wiki/File:Location_map_Fukui_citywide.svg [17:14:02] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393457 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [17:14:34] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393458 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [17:14:50] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393459 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [17:49:13] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393663 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [17:53:56] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393679 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [17:59:52] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393689 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [18:09:09] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393718 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [18:09:41] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393722 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [18:10:00] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393724 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [18:11:37] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393740 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [18:13:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11393756 (10RobH) Day 8 Update: * 22 hosts moved today, 22 remain ** all wikikube and aux host migrations completed ** (3) pc hosts in disucssion with data-p... [18:14:08] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393760 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [18:14:17] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393761 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [18:54:25] FIRING: SystemdUnitFailed: bird.service on hcaptcha-proxy7002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:54:31] yeah ok silencing, reimaging soon [18:59:58] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393865 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [19:02:41] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393870 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [19:05:03] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393873 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [19:08:47] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393880 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [19:09:25] FIRING: [2x] SystemdUnitFailed: bird.service on hcaptcha-proxy7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:16:44] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393895 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [19:16:44] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393896 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [19:16:50] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393898 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [19:16:57] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393899 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was st... [19:25:08] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393914 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [19:29:18] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11393921 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [20:07:36] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11394023 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [20:12:47] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11394043 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [20:15:46] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11394044 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [20:22:26] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: Measure request frequency of thumbnail sizes - https://phabricator.wikimedia.org/T410304#11394056 (10Ladsgroup) Very likely a popular gadget/css hardcoding the url. I investigate once I get my hands on a PC [20:24:35] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11394057 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage starte... [20:27:34] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests, 13Patch-For-Review: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11394058 (10ssingh) 05Open→03Resolved a:03ssingh Once https://ger...