[03:08:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp6008:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [03:09:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp6008 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=drmrs&var-instance=cp6008 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [03:55:48] Not sure what's going on with it but it's not responding. Depooed [03:57:34] memory error, yay! [04:04:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp6008 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=drmrs&var-instance=cp6008 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [04:05:46] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12102916 (10BCornwall) [04:13:21] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12102935 (10BCornwall) [08:28:09] FIRING: LVSHighRX: Excessive RX traffic on lvs3009:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs3009 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [08:43:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs3009:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs3009 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [09:23:09] 10netops, 06Infrastructure-Foundations, 06SRE: Faulty port cr2-eqord:xe-0/1/1 - https://phabricator.wikimedia.org/T252988#12103639 (10cmooney) 05Open→03Resolved a:03cmooney Port is currently not in use, router seems healthy. [09:26:03] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12103661 (10cmooney) [09:32:13] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#12103725 (10cmooney) [09:36:25] 10netops, 06Infrastructure-Foundations, 06SRE: Cookbook sre.network.configure-switch-interfaces failing on upgraded Juniper switch - https://phabricator.wikimedia.org/T428071#12103739 (10ayounsi) 05Open→03Resolved [09:38:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch upgrade - https://phabricator.wikimedia.org/T418012#12103752 (10cmooney) p:05Medium→03High [09:38:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12103758 (10cmooney) [09:45:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12103797 (10cmooney) p:05Low→03Medium [09:52:23] 10netops, 06Infrastructure-Foundations, 06SRE: OSPF metrics - https://phabricator.wikimedia.org/T200277#12103837 (10ayounsi) 05Open→03Declined Not relevant soon with the new transport links. [09:52:35] 10netops, 06Infrastructure-Foundations, 06SRE: Cleanup confed BGP peerings and policies - https://phabricator.wikimedia.org/T167841#12103839 (10cmooney) 05Open→03Declined This work is going to be done, but we are tracking the overall WAN configuration in task T297355 we don't need this separate one. [10:15:22] 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup), 07Wikimedia-Incident: Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683 (10MatthewVernon) 03NEW [10:21:19] 10netops, 06Infrastructure-Foundations, 06SRE: Create alerting for saturation on sub-rated interfaces - https://phabricator.wikimedia.org/T374614#12104022 (10ayounsi) a:03ayounsi [10:28:04] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox network report failing - timeout errors - https://phabricator.wikimedia.org/T321704#12104049 (10cmooney) 05Open→03Resolved Closing this task, the issue was not in the test_matching_vlan() function. Thankfully right now we do... [10:30:05] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-Needs-Improvement: test_matching_vlan() function crashing in Netbox network report - https://phabricator.wikimedia.org/T339133#12104051 (10cmooney) 05Open→03Resolved a:03cmooney [10:38:49] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia: how to approach schema differences in SR-Linux versions - https://phabricator.wikimedia.org/T412157#12104102 (10cmooney) 05Open→03Resolved a:03cmooney Gonna close this for now. We've tackled it by just having different files that get added for... [11:58:23] 06Traffic, 10RESTBase, 10RESTBase-Sunsetting, 06serviceops-radar, and 2 others: Block traffic to RESTBase /page/related endpoint and sunset it - https://phabricator.wikimedia.org/T376297#12104650 (10MSantos) [12:14:34] 06Traffic, 10Page Content Service, 06Wikipedia-Android-App-Backlog, 06Content-Transform-Team (Work In Progress), 07Essential-Work: [[2025 Coeur d'Alene shooting]] showing old version in Android app - https://phabricator.wikimedia.org/T398243#12104780 (10MSantos) [12:16:13] 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup), 07Wikimedia-Incident: Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683#12104805 (10ayounsi) [12:16:19] 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup), 07Wikimedia-Incident: Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683#12104807 (10ayounsi) a:03ayounsi [12:26:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12104878 (10VRiley-WMF) @cmooney Yes, the majority of them should be done, but I'll be double checking them as well. A8 willl need more time for the racking.... [13:29:49] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q1 FY2026-27), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12105235 (10BWojtowicz-WMF) I removed the `DTYPE=float16` and exchanged it with `DTYPE=auto` env var and service still responds happily,... [14:15:48] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12105447 (10cmooney) The patch is merged, and we can see the local-pref=100 on standard routes being learnt now: ` A:ssw1-d8-e... [14:32:46] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Wikimedia-Performance-recommendation: Change the webp threshold based on access distribution - https://phabricator.wikimedia.org/T431150#12105589 (10JTweed-WMF) [14:32:50] 06Traffic, 06Data-Persistence, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12105590 (10JTweed-WMF) [14:35:49] 06Traffic, 06SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334#12105630 (10JTweed-WMF) [14:58:28] 06Traffic, 06SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334#12105821 (10JTweed-WMF) [14:58:35] 06Traffic, 06Data-Persistence, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12105818 (10JTweed-WMF) [15:25:59] 10netops, 06Infrastructure-Foundations, 06SRE: Renumber link addressing to lsw1-e1-eqiad and lsw1-f1-eqiad - https://phabricator.wikimedia.org/T404248#12105978 (10cmooney) 05Open→03Resolved This was resolved some time back. [15:27:53] 06Traffic, 06Security-Team, 072026-user-javascript-incident, 07ContentSecurityPolicy, 07SecTeam-Processed: Can't debug scripts on localhost on URLs that omit /w/index.php - https://phabricator.wikimedia.org/T421565#12105987 (10sbassett) a:03sbassett [17:13:15] 10netops, 06Infrastructure-Foundations, 06SRE: Export network fixed-limit usage stats in gnmi - https://phabricator.wikimedia.org/T395998#12106342 (10cmooney) [17:14:02] 10netops, 06Infrastructure-Foundations, 06SRE: Export network fixed-limit usage stats in gnmi - https://phabricator.wikimedia.org/T395998#12106344 (10cmooney) The first of these is tracked in T428685 for Nokia devices. Leaving this one open for now to do the work for Juniper. [17:18:24] 10netops, 06Traffic, 10Cloud-Services, 06collaboration-services, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12106350 (10cmooney) [17:24:24] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745 (10cmooney) 03NEW p:05Triage→03High [17:25:22] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106371 (10cmooney) [17:25:52] 10netops, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Nokia SR-Linux DHCP Relay Bug - https://phabricator.wikimedia.org/T411054#12106375 (10cmooney) [17:25:53] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106376 (10cmooney) [17:26:41] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad: move row-wide vlan gateways to Nokia switches - https://phabricator.wikimedia.org/T416872#12106378 (10cmooney) [17:26:42] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106379 (10cmooney) [17:26:59] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106381 (10cmooney) [17:27:00] 10netops, 06Infrastructure-Foundations, 06SRE: Eqiad C/D refresh: move legacy switch uplinks to Nokias and migrate Vlan GWs - https://phabricator.wikimedia.org/T405562#12106382 (10cmooney) [17:27:01] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux - wonky routing with IPv6 RAs and EVPN Anycast GW - https://phabricator.wikimedia.org/T420706#12106380 (10cmooney) [17:27:22] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106384 (10cmooney) [17:27:23] 10netops, 06Infrastructure-Foundations, 06SRE: Blackbox probe for TLS cert expiry failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12106383 (10cmooney) [17:27:32] 10netops, 06Infrastructure-Foundations: Add (some) collection for Nokia SR-Linux components - https://phabricator.wikimedia.org/T428685#12106386 (10cmooney) [17:27:35] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106387 (10cmooney) [17:27:56] 10netops, 06Infrastructure-Foundations, 06SRE: SR-Linux: applying analytics-in acl to irb sub-interface blocks ARP - https://phabricator.wikimedia.org/T429499#12106388 (10cmooney) [17:27:57] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12106389 (10cmooney) [17:39:22] 10netops, 06Infrastructure-Foundations, 06SRE: Create netbox script to support moving a cable from one network port to another - https://phabricator.wikimedia.org/T355869#12106433 (10cmooney) 05Open→03Declined We've done without it thus far ok. If we need it again we can revisit. [17:45:58] 10netops, 06Infrastructure-Foundations, 06SRE: Core router upgrades 2026 #2 - https://phabricator.wikimedia.org/T431748 (10cmooney) 03NEW p:05Triage→03Medium