[03:36:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12428727 (10Papaul) [10:21:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12429830 (10Jclark-ctr) Installed 4x plr4 optics in spines [11:01:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12429998 (10Jclark-ctr) Updated the remaining links between E/F spines to A1/A8 spine in netbox [13:34:18] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12430840 (10Fabfur) Confirmed that the cache servers are now returning the `deprecation` and `sunset` headers for that path [13:34:41] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12430842 (10Fabfur) [13:51:53] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12431081 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=44b71149-ad3f-4bc3-a181-1d317af25c78) set by pt1979@cumin1004 for 2:00:00 on 8 host(s) and their services with reason: J... [13:54:26] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12431090 (10Clement_Goubert) ` curl -v https://en.wikipedia.org/api/rest_v1/media/math/ 2>&1| grep -E 'deprecation|sunset' < deprecation: @17881... [15:13:42] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12431944 (10Papaul) [15:19:15] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#12431997 (10Papaul) 05Open→03Resolved This is complete [15:19:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12432000 (10Papaul) [15:25:34] 06Traffic: revisit disabling virtualization on Dell hosts - https://phabricator.wikimedia.org/T441020#12432045 (10ssingh) [15:53:55] 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations, 13Patch-For-Review: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#12432159 (10elukey) 05Open→03Resolved [16:54:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp2044:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=codfw%20prometheus/ops&var-instance=cp2044&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [16:54:43] FIRING: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2044:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2044 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [16:55:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2044 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2044 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [16:59:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp2044:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=codfw%20prometheus/ops&var-instance=cp2044&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [16:59:43] RESOLVED: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2044:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2044 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [17:00:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2044 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2044 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [17:09:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2044 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2044 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [17:11:02] cjd91: are you still working on this? [17:14:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2044 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2044 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [17:24:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2044:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [17:24:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2044 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2044 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [18:12:27] cjd91: cp2044 is pooled but still downtimed [18:12:31] please let's remove the downtiming on that [18:22:55] I went ahead and removed it [18:25:03] thanks [18:25:15] cjd91: please check your notifications for this channel :> [18:30:28] got it. I'll keep a close eye on this channel when doing any operational work [19:10:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2045:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [19:11:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2045 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2045 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:34:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2045:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [19:39:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2045:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [19:39:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2045 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2045 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [20:16:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp2046:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=codfw%20prometheus/ops&var-instance=cp2046&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [20:16:43] FIRING: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2046:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2046 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [20:21:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp2046:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=codfw%20prometheus/ops&var-instance=cp2046&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [20:21:43] RESOLVED: HaproxyKafkaRestarted: HaproxyKafka restarted on cp2046:9100 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaRestarted - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2046 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaRestarted [20:27:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp2046 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2046 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [20:28:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2046:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [20:33:40] RESOLVED: VarnishPrometheusExporterDown: Varnish Exporter on instance cp2046:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [20:37:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp2046 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=codfw&var-instance=cp2046 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown