[06:13:15] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#12404668 (10ayounsi) ulsfo too, with {T434068} [07:20:04] 06Traffic, 06Collaboration-Services, 06SRE: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12404758 (10LSobanski) [09:21:27] moritzm: did you restart pybal in eqiad after merging the ldap service CR? [09:23:16] yes, on lvs1018/1020, see https://phabricator.wikimedia.org/T439730 for the times [09:23:42] did I miss anything? [09:24:05] lvs1018 is complaining about pybal not being restarted [09:24:30] and systemd agrees: Active: active (running) since Thu 2026-08-06 14:09:35 UTC; 2 months 0 days ago [09:25:19] hmmh,let me check [09:26:11] hmmh, seems I infact missed that. doing it now [09:26:16] ack, thx [09:27:38] done and alert has cleared, sorry for the omission [09:27:55] no problem [09:46:55] 06Traffic, 06SRE, 10WikimediaDebug: Debug server routing does not work for /w/rest.php - https://phabricator.wikimedia.org/T440439 (10daniel) 03NEW [10:16:24] 06Traffic: reload-vcl fails intermittently on cp nodes - https://phabricator.wikimedia.org/T440441 (10Vgutierrez) 03NEW [10:16:45] 06Traffic: reload-vcl fails intermittently on cp nodes - https://phabricator.wikimedia.org/T440441#12405389 (10Vgutierrez) p:05Triage→03Medium [11:43:31] 06Traffic, 06SRE, 10WikimediaDebug: Debug server routing does not work for /w/rest.php - https://phabricator.wikimedia.org/T440439#12405731 (10Vgutierrez) `backend=k8s-mwdebug` isn't currently allowed for `/w/rest.php`: `lang=lua -- For rest.php, only allow debug remapping for mw-experimental and mw-pars... [11:54:51] I might have broken cp hosts by removing the graphite config [11:57:40] Weirdly enough not all of them [11:58:47] gonna look for another minute or two but I can roll back whenever - I think reverting the bits that touch varnish/trafficserver config will be enough though [11:59:00] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1332767 this is the change [11:59:39] slyngs: is that just a case of them not having run puppet yet? [12:00:17] Up, it's probably "just" the one cluster [12:02:27] Yeah, okay, so text is broken, upload isn't [12:03:16] prepping the revert [12:07:34] oh wait, could this be vgutierrez's change? [12:07:59] https://gerrit.wikimedia.org/r/1352477 [12:09:47] yeah it is [12:11:49] Oooh: TypeError: sequence item 2: expected str instance, int found [12:12:03] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352495 [12:12:07] Yeah, it might not love that the timeout is parsed as an int [12:12:23] funny coincidence that the change went out and then I immediately hit it [12:12:29] by causing the reload [12:13:10] We can probably find a way to blame you if need be :-) [12:14:13] Let's try triggering a puppet run on one of the hosts when the patch i merged [12:17:09] running [12:18:00] I've started a run on cp7007. [12:19:41] looks like it works [12:19:55] phew [12:19:59] :-) [12:20:21] cp7007 is happy [12:21:56] I think this: "Exec[retry-load-new-vcl-file-frontend] success notrun ["0"]" in Puppetboard means that reload ran [12:54:51] ouch, thx hnowlan <3 [13:43:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12406199 (10cmooney) >>! In T418018#12403321, @Jclark-ctr wrote: > @cmooney @RobH Finished A8 LSW, spine management, and SCS cabling. Updated the SCS... [13:45:33] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406205 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b92fad2c-3918-4020-9934-ce2871a5d804) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [14:21:30] FIRING: [2x] LibericaStaleConfig: Liberica instance lvs3008 is running a stale configuration - https://wikitech.wikimedia.org/wiki/Liberica#LibericaStaleConfig - https://alerts.wikimedia.org/?q=alertname%3DLibericaStaleConfig [14:21:44] ^^ that's me and it's expected [14:26:30] FIRING: LibericaStaleConfig: Liberica instance lvs7002 is running a stale configuration - https://wikitech.wikimedia.org/wiki/Liberica#LibericaStaleConfig - https://grafana.wikimedia.org/d/fa4de97a-7114-48c7-a91a-f56089ef554f/liberica?orgId=1&viewPanel=10&var-site=magru&var-instance=lvs7002 - https://alerts.wikimedia.org/?q=alertname%3DLibericaStaleConfig [14:29:44] I wonder why that fired given the alert is acknowledged :) [14:31:30] FIRING: [4x] LibericaStaleConfig: Liberica instance lvs6001 is running a stale configuration - https://wikitech.wikimedia.org/wiki/Liberica#LibericaStaleConfig - https://alerts.wikimedia.org/?q=alertname%3DLibericaStaleConfig [14:36:30] FIRING: [11x] LibericaStaleConfig: Liberica instance lvs3009 is running a stale configuration - https://wikitech.wikimedia.org/wiki/Liberica#LibericaStaleConfig - https://alerts.wikimedia.org/?q=alertname%3DLibericaStaleConfig [14:41:30] RESOLVED: [3x] LibericaStaleConfig: Liberica instance lvs4009 is running a stale configuration - https://wikitech.wikimedia.org/wiki/Liberica#LibericaStaleConfig - https://alerts.wikimedia.org/?q=alertname%3DLibericaStaleConfig [15:03:46] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406857 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=973a1f0e-7b13-4b84-a662-c464cd8b1c39) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:04:02] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406858 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=3dad56c8-3d29-4732-8cdc-888cbff5cd56) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:05:00] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406878 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5b1d769e-793d-41ab-9f6d-8ffbbf1b0f05) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:05:30] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406881 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=84bef5d3-d308-4bff-9956-cd6ef184cc99) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:06:24] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406887 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8edef278-f04e-4382-b245-c0737e83964a) set by pt1979@cumin1004 for 2:00:00 on 1 host(s) and their services with reason: J... [15:07:07] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406898 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=06d6d295-b237-4905-b49b-224383a6da43) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:07:33] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406900 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=559e6f53-818c-4947-be1b-1e17f06db0ca) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:08:25] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406912 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=08788b61-9f61-4307-acc1-9beec96b1b26) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:09:45] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406934 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1dc6ec98-e477-4fe8-b41e-e03982222d6b) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:10:49] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406938 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8457b4c5-c815-4920-9fe5-0c1a08027db8) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:11:12] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12406942 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7926f577-2f16-46c7-8af4-b515985fe43b) set by pt1979@cumin1004 for 2:00:00 on 2 host(s) and their services with reason: J... [15:37:47] 06Traffic, 06ServiceOps, 10Wikidata, 06Wikidata Platform Team, and 3 others: Migrate the deployment of WDQS UI from wikikube to dse-k8s - https://phabricator.wikimedia.org/T436224#12407134 (10Gehel) [15:48:02] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12407308 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cc8c6577-fecc-488f-ba8f-baf0394868ba) set by pt1979@cumin1004 for 1:00:00 on 3 host(s) and their services with reason: J... [15:48:32] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12407313 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9feb482a-36e5-4ef0-bd31-e2381cbba1be) set by pt1979@cumin1004 for 2:00:00 on 3 host(s) and their services with reason: J... [15:51:14] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12407331 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ef3a779c-6b51-41af-927e-15706021fa93) set by cmooney@cumin1004 for 2:00:00 on 4 host(s) and their services with reason:... [17:01:49] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12408084 (10Papaul) [17:02:49] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12408094 (10Papaul) Cr2-codfw Junos upgrade finished. Tout many steps to move traffic away from the router. Thanks @cmooney for the tips [18:09:10] 10netops, 06Infrastructure-Foundations: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12408474 (10wiki_willy) To close the loop, we received a credit back for $269.17 [21:18:32] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12409116 (10ArthurPSmith) Note maxlag continues to be bad for much of the time, preventing bot edits. I see this thread is mostly about pywikibot and reads;...