[00:09:25] RESOLVED: SystemdUnitFailed: wmf_auto_restart_varnish-frontend-fetcherr.service on cp3074:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:50:04] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12423824 (10ayounsi) [07:50:05] 10netops, 06Infrastructure-Foundations: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563#12423825 (10ayounsi) [09:20:04] 06Traffic, 10hCaptcha, 06Product Safety and Integrity, 06ServiceOps, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12424073 (10MLechvien-WMF) 05Open→03Resolved Resolving as the fix deployed in T420223 seems to have improved the situation.... [09:28:32] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424097 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d0aeef7c-1051-4ca5-b9ec-22a2953df15a) set by cmooney@cumin1004 for 1:00:00 on 18 host(s) and their servic... [09:29:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12424104 (10ayounsi) We should probably try to cross check devices: * Connect a mgmt port showing the issue on a neighbor switch (te... [09:47:02] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424186 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=15f83802-d41c-4de8-9b3f-978d7aa13ee8) set by cmooney@cumin1004 for 1:00:00 on 18 host(s) and their servic... [10:03:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12424279 (10cmooney) >>! In T440631#12424104, @ayounsi wrote: > We should probably try to cross check devices: > * Reboot the mgmt s... [10:08:01] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424315 (10cmooney) [10:10:11] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12424333 (10cmooney) I've made a start on this tackling //ssw1-d1-eqiad// first, and will do the same on //ssw1-d8-eqiad// in a few days if all seems ok. Reaso... [14:10:03] Hi, a rather small change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1354791 [14:14:12] Amir1: interesting [15:09:05] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimdeia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990 (10cmooney) 03NEW p:05Triage→03Medium [15:09:21] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimdeia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12425814 (10cmooney) [15:09:32] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07User-notice: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12425815 (10cmooney) [15:13:41] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12425836 (10Nirmos) [15:16:26] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12425864 (10cmooney) [15:40:29] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999 (10cmooney) 03NEW p:05Triage→03Low [15:40:39] Hello - little bit of monitoring cleanup if anyone has a few minutes to review. These checks are superfluous and we already get alerts via the systemd checks [15:40:48] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344227 (I realise there's some non-traffic stuff in here too, sorry) [15:40:55] https://gerrit.wikimedia.org/r/1344231 [16:42:49] hnowlan: Thanks for those patches. As I wrote in one of them, I'm a little concerned about deploying those at this moment since I think icinga checks are (unfortunately) a big part of ensuring healthiness during cookbook runs. I wonder if this is a good time for us to work towards modifying various cookbooks to verify that no AM alerts are firing that match a specific criteria [16:43:04] That way we can reomve such reliance on icinga. [16:45:23] aha that's a very good point [17:12:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014 (10cmooney) 03NEW p:05Triage→03High [17:12:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12426764 (10cmooney) [17:12:41] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12426765 (10cmooney) [17:22:49] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12426846 (10RobH) Cathal and I chatted about this in IRC so I'm summarizing my points here for task visibility. I don't really wanna slow d... [17:29:11] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12426889 (10RobH) Edit addition. I understand this a bit better, it would be mainly for the 85 wikikube hosts in rows A/B 1) new 10G conne... [17:38:38] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: modify move-server script to allow for 1G->10G upgrade during process - https://phabricator.wikimedia.org/T440999#12426949 (10cmooney) >>! In T440999#12426889, @RobH wrote: > I understand this a bit better, it would be mainly for the 85 wikikube hosts in... [18:15:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12427147 (10cmooney) [18:27:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12427186 (10cmooney) [18:38:45] 06Traffic: Remove OCSP support from acme-chief - https://phabricator.wikimedia.org/T441025 (10CDobbins) 03NEW [18:56:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2043:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [18:57:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2043 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2043 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:00:18] cjd91: ^ we should repool this I think given the virtualization thing didn't work out? [19:00:26] with the usual checks [19:00:52] hmm it's not even up ok [19:00:59] so yeah, let's bring it back up and pool, or downtime if you are working on it [19:11:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2043:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [19:17:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2043 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2043 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:22:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2043:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [19:23:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2043 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2043 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:27:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2043:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [19:28:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2043 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2043 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [20:12:20] went ahead and downtimed it for 1D [20:17:48] thanks