[01:56:40] FIRING: SystemdUnitFailed: fetch-rings-thanos.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:56:40] FIRING: SystemdUnitFailed: fetch-rings-thanos.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:59:29] ^-- should be fixed now [07:01:25] RESOLVED: SystemdUnitFailed: fetch-rings-thanos.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:13:49] thanos-be1008 also needed re-provisioning before it could UEFI HTTP boot. Is this bug likely to impact all the ms-be* nodes as well as thanos-be*? These hosts have all been reimaged OK in the past [but if it's all of them and/or there was some way to tell without trying a reimage and waiting 10 minutes for it to fail, that would speed up the reimaging process somewhat] [07:20:44] Emperor: not your answer, but would it help to be able to chain both cookbooks in an unattended way? like a --provision parameter to the reimage cookbook? [07:21:27] Emperor: was the previous re-image over UEFI HTTP? [07:27:09] Emperor: btw, 3 ms-be will be unreachable for ~15min in 30min - https://phabricator.wikimedia.org/T430910 [07:33:34] XioNoX: these nodes were all UEFI reimaged before, but they might previously have been done via PXE? I dunno when we moved [07:34:05] B4> ack, thanks for the reminder, LMK when done [07:35:33] If all the backends are going to need reprovisioning, then it's not hard to change the process to take that into account (and that will save some time); but if mostly they don't, then it's obviously better not to :) [09:12:37] Emperor: in theory if they already UEFI-reimaged you shouldn't see issues, these thanos-be ones may have been misconfigured somehow [09:13:30] elukey: they were all previously UEFI-imaged (when originally purchased). [09:23:09] 10netops, 06Infrastructure-Foundations, 06SRE: Faulty port cr2-eqord:xe-0/1/1 - https://phabricator.wikimedia.org/T252988#12103639 (10cmooney) 05Open→03Resolved a:03cmooney Port is currently not in use, router seems healthy. [09:26:03] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12103661 (10cmooney) [09:32:11] Emperor: yeah so they used HTTP boot [09:32:13] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#12103725 (10cmooney) [09:36:25] 10netops, 06Infrastructure-Foundations, 06SRE: Cookbook sre.network.configure-switch-interfaces failing on upgraded Juniper switch - https://phabricator.wikimedia.org/T428071#12103739 (10ayounsi) 05Open→03Resolved [09:38:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch upgrade - https://phabricator.wikimedia.org/T418012#12103752 (10cmooney) p:05Medium→03High [09:38:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12103758 (10cmooney) [09:45:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12103797 (10cmooney) p:05Low→03Medium [09:52:23] 10netops, 06Infrastructure-Foundations, 06SRE: OSPF metrics - https://phabricator.wikimedia.org/T200277#12103837 (10ayounsi) 05Open→03Declined Not relevant soon with the new transport links. [09:52:35] 10netops, 06Infrastructure-Foundations, 06SRE: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841#12103839 (10cmooney) 05Open→03Declined This work is going to be done, but we are tracking the overall WAN configuration in task T297355 we don't need this separate one. [10:15:22] 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup), 07Wikimedia-Incident: Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683 (10MatthewVernon) 03NEW [10:21:19] 10netops, 06Infrastructure-Foundations, 06SRE: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#12104022 (10ayounsi) a:03ayounsi [10:28:04] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox network report failing - timeout errors - https://phabricator.wikimedia.org/T321704#12104049 (10cmooney) 05Open→03Resolved Closing this task, the issue was not in the test_matching_vlan() function. Thankfully right now we do... [10:30:05] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-Needs-Improvement: test_matching_vlan() function crashing in Netbox network report - https://phabricator.wikimedia.org/T339133#12104051 (10cmooney) 05Open→03Resolved a:03cmooney [10:38:47] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia: how to approach schema differences in SR-Linux versions - https://phabricator.wikimedia.org/T412157#12104102 (10cmooney) 05Open→03Resolved a:03cmooney Gonna close this for now. We've tackled it by just having different files that get added for... [10:57:43] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [10:58:09] ^ that is me,on it [12:16:11] 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup), 07Wikimedia-Incident: Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683#12104805 (10ayounsi) [12:16:17] 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup), 07Wikimedia-Incident: Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683#12104807 (10ayounsi) a:03ayounsi [12:26:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12104878 (10VRiley-WMF) @cmooney Yes, the majority of them should be done, but I'll be double checking them as well. A8 willl need more time for the racking.... [14:15:48] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12105447 (10cmooney) The patch is merged, and we can see the local-pref=100 on standard routes being learnt now: ` A:ssw1-d8-e... [14:57:43] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [15:25:59] 10netops, 06Infrastructure-Foundations, 06SRE: Renumber link addressing to lsw1-e1-eqiad and lsw1-f1-eqiad - https://phabricator.wikimedia.org/T404248#12105978 (10cmooney) 05Open→03Resolved This was resolved some time back. [17:13:15] 10netops, 06Infrastructure-Foundations, 06SRE: Export network fixed-limit usage stats in gnmi - https://phabricator.wikimedia.org/T395998#12106342 (10cmooney) [17:14:02] 10netops, 06Infrastructure-Foundations, 06SRE: Export network fixed-limit usage stats in gnmi - https://phabricator.wikimedia.org/T395998#12106344 (10cmooney) The first of these is tracked in T428685 for Nokia devices. Leaving this one open for now to do the work for Juniper. [17:18:23] 10netops, 10Cloud-Services, 06collaboration-services, 06Data-Persistence, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12106350 (10cmooney) [17:24:23] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745 (10cmooney) 03NEW p:05Triage→03High [17:25:21] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106371 (10cmooney) [17:25:52] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#12106375 (10cmooney) [17:25:53] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106376 (10cmooney) [17:26:41] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad: move row-wide vlan gateways to Nokia switches - https://phabricator.wikimedia.org/T416872#12106378 (10cmooney) [17:26:42] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106379 (10cmooney) [17:26:58] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106381 (10cmooney) [17:26:59] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562#12106382 (10cmooney) [17:27:01] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux - wonky routing with IPv6 RAs and EVPN Anycast GW - https://phabricator.wikimedia.org/T420706#12106380 (10cmooney) [17:27:22] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106384 (10cmooney) [17:27:23] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expiry failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12106383 (10cmooney) [17:27:32] 10netops, 06Infrastructure-Foundations: Add (some) collection for Nokia SR-Linux components - https://phabricator.wikimedia.org/T428685#12106386 (10cmooney) [17:27:34] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106387 (10cmooney) [17:27:56] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux: applying analytics-in acl to irb sub-interface blocks ARP - https://phabricator.wikimedia.org/T429499#12106388 (10cmooney) [17:27:57] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106389 (10cmooney) [17:39:22] 10netops, 06Infrastructure-Foundations, 06SRE: Create netbox script to support moving a cable from one network port to another - https://phabricator.wikimedia.org/T355869#12106433 (10cmooney) 05Open→03Declined We've done without it thus far ok. If we need it again we can revisit. [17:45:58] 10netops, 06Infrastructure-Foundations, 06SRE: Core router upgrades 2026 #2 - https://phabricator.wikimedia.org/T431748 (10cmooney) 03NEW p:05Triage→03Medium [18:57:43] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [22:57:43] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed