[07:33:22] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12435409 (10MLechvien-WMF) 05In progress→03Resolved Thanks Fabrizio and Clement! Marking this as resolved, @HCoplin-WMF let us know if a... [09:09:45] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Viewing image in browser tab produces chain of errors (thumb.wikimedia.org lacks a favicon) - https://phabricator.wikimedia.org/T436577#12435756 (10JTweed-WMF) p:05Triage→03High [12:08:16] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12436461 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=945bd817-b2ad-49ca-aa6a-15d15e909727) set by cmooney@cumin1004 for 1:00:00 on 19 host(s) and their servic... [15:09:58] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12437505 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6c656e79-99d3-44aa-af65-81b384925d60) set by pt1979@cumin1004 for 2:00:00 on 14 host(s) and their services with reas... [15:14:14] 06Traffic, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07Regression: Viewing image in browser tab produces chain of errors (thumb.wikimedia.org lacks a favicon) - https://phabricator.wikimedia.org/T436577#12437520 (10Vgutierrez) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1355652 adds htt... [15:19:30] 10netops, 06Infrastructure-Foundations, 06SRE: eqiad: pod CD switch upgrade (2026) - https://phabricator.wikimedia.org/T431745#12437567 (10cmooney) [15:22:47] 06Traffic, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07Regression: Viewing image in browser tab produces chain of errors (thumb.wikimedia.org lacks a favicon) - https://phabricator.wikimedia.org/T436577#12437590 (10Ladsgroup) yeah, I can work on that, we probably need to carve out an exception i... [15:47:32] 06Traffic, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review, 07Regression: Viewing image in browser tab produces chain of errors (thumb.wikimedia.org lacks a favicon) - https://phabricator.wikimedia.org/T436577#12437729 (10Vgutierrez) yes... that seems to be enough, given I get a proper favicon if a byp... [15:50:33] Amir1: I can take care of merging those 2 CRs together and save a VCL reload cycle :) [16:04:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12438000 (10Papaul) [16:04:25] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12438001 (10Papaul) [16:07:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12438024 (10Papaul) a:05Papaul→03cmooney [16:21:23] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12438137 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=4e90d5a0-08fa-429f-892b-181bf8b8f381) set by pt1979@cumin2003 for 0:05:00 on 8 host(s) and their services with reaso... [16:40:12] vgutierrez: if you're still around, go for it. I'm running an errand. Can do both once I'm back if you called it a day [16:40:22] ack [17:10:32] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12438471 (10Vgutierrez) I'm just double checking the service catalog. We have 102 services using IPIP encapsulation: ` ~/wikimedia.or... [17:24:13] 10netops, 06Infrastructure-Foundations, 06SRE: Cannot send IPv6 packets to device on private1-b3-eqiad - https://phabricator.wikimedia.org/T441350 (10cmooney) 03NEW p:05Triage→03High [17:24:34] 10netops, 06Infrastructure-Foundations, 06SRE: Cannot send IPv6 packets to device on private1-b3-eqiad - https://phabricator.wikimedia.org/T441350#12438531 (10cmooney) [17:24:41] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12438532 (10cmooney) [17:25:15] 10netops, 06Infrastructure-Foundations, 06SRE: Cannot send IPv6 packets to device on private1-b3-eqiad - https://phabricator.wikimedia.org/T441350#12438535 (10cmooney) [17:27:18] 10netops, 06Infrastructure-Foundations, 06SRE: Cannot send IPv6 packets to device on private1-b3-eqiad - https://phabricator.wikimedia.org/T441350#12438544 (10cmooney) [17:35:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2047:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [17:35:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2047 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2047 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [17:40:39] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12438665 (10cmooney) >>! In T440859#12438471, @Vgutierrez wrote: >it looks like we have 3 services without IPIP encapsulation: Thank... [17:40:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2047:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [17:40:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2047 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2047 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [17:52:33] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12438699 (10ssingh) >>! In T440859#12438665, @cmooney wrote: >>>! In T440859#12438471, @Vgutierrez wrote: >>it looks like we have 3 s... [17:57:40] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, 07Kubernetes: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12438719 (10ssingh) 05Resolved→03Open Sorry for re-opening this folks but it seems like Sophroid is still on IPIP and was not tracked a... [18:01:06] 06Traffic, 10Lift-Wing: Migrate inference and inference-staging to IPIP - https://phabricator.wikimedia.org/T441352 (10ssingh) 03NEW [18:02:12] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12438741 (10ssingh) Filed T441352 for `inference` and `inference-staging` move and T441352 for Sophroid. [18:12:05] 10netops, 06Infrastructure-Foundations, 06SRE: Cannot send IPv6 packets to device on private1-b3-eqiad - https://phabricator.wikimedia.org/T441350#12438758 (10cmooney) Spent a bit more time looking at this. Still nothing jumping out at me, I'll have to open a ticket with Nokia tomorrow unless @ayounsi can s... [18:26:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2048 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2048 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [18:31:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2048 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2048 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [18:39:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2048:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [18:39:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2048 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2048 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [18:40:50] 10netops, 06Infrastructure-Foundations, 06SRE: Cannot send IPv6 packets to device on private1-b3-eqiad - https://phabricator.wikimedia.org/T441350#12438869 (10cmooney) 05Open→03Resolved So... I laughed hard when I realised what this was :D I had manually set the IPs on sretest1006 after dc-ops moved... [18:44:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2048:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [18:49:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2048 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2048 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:00:28] 06Traffic, 07Sustainability (Incident Followup): alert on server_ip NEL field for each DNS-pooled PoP - https://phabricator.wikimedia.org/T426301#12438965 (10ssingh) @CDanis: This came up during an operational review meeting today for tasks in the backlog for incident followup. Two quick questions that could u... [20:09:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2049 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2049 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [20:14:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2049 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2049 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [20:28:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2049:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [20:28:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2049 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2049 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [20:30:10] the amount of noise that scheduled maintenance triggers is ridiculous [20:33:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2049:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [20:38:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2049 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2049 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [21:09:29] yeah they are. and they should not [21:09:40] will file a task [21:24:26] 06Traffic, 07Sustainability (Incident Followup): alert on server_ip NEL field for each DNS-pooled PoP - https://phabricator.wikimedia.org/T426301#12439621 (10CDanis) >>! In T426301#12438964, @ssingh wrote: > 1. Do you think it makes sense to do this? I think it does but wanted to check. Yes, definitely. I di... [21:29:49] 06Traffic, 07Sustainability (Incident Followup): alert on server_ip NEL field for each DNS-pooled PoP - https://phabricator.wikimedia.org/T426301#12439642 (10CDanis) @hnowlan btw I'm very interested in any o11y team takes on the inhibition rules thoughts [22:51:07] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Viewing image in browser tab produces chain of errors (thumb.wikimedia.org lacks a favicon) - https://phabricator.wikimedia.org/T436577#12439933 (10Krinkle) 05Open→03Resolved a:03Vgutierrez