[03:05:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp1104:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:09:40] FIRING: [4x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp1108:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [03:09:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp1111 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=eqiad&var-instance=cp1111 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [03:09:47] FIRING: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp1108 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [03:10:00] FIRING: PurgedHighEventLag: High event process lag with purged on cp7004:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7004 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [03:21:25] RESOLVED: [4x] VarnishPrometheusExporterDown: Varnish Exporter on instance cp1108:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [03:21:29] RESOLVED: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp1109 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [03:21:33] RESOLVED: [2x] HaproxyKafkaExporterDown: HaproxyKafka on cp1108 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [03:21:52] RESOLVED: [2x] PurgedHighEventLag: High event process lag with purged on cp7004:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7004 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [03:22:06] RESOLVED: [2x] VarnishHighThreadCount: Varnish's thread count on cp1104:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:43:51] 06Traffic, 10MobileFrontend (Tracking): All Wikimedia projects provides mobile frontend on Huawei Laptops running HarmonyOS - https://phabricator.wikimedia.org/T408567#11361790 (10Krinkle) >>! In T408567#11319789, @Jdlrobson-WMF wrote: > Relevant code is in puppet https://gerrit.wikimedia.org/g/operations/pupp... [04:07:21] 06Traffic, 06MediaWiki-Platform-Team (Radar), 07Upstream: Telegram previews broken since unified mobile routing - https://phabricator.wikimedia.org/T409575#11361802 (10Krinkle) Telegram's link preview service does follow redirects for other websites. I tried it on my personal domain, and later on people.wiki... [05:23:46] 10netops, 06Infrastructure-Foundations, 06SRE: Row C traffic outage Nov 11 2025 - https://phabricator.wikimedia.org/T409800 (10cmooney) 03NEW p:05Triage→03High [05:53:02] 10netops, 06Infrastructure-Foundations, 06SRE: Row C traffic outage Nov 11 2025 - https://phabricator.wikimedia.org/T409800#11361879 (10cmooney) [05:58:32] 10netops, 06Infrastructure-Foundations, 06SRE: Row C traffic outage Nov 11 2025 - https://phabricator.wikimedia.org/T409800#11361886 (10cmooney) [06:48:43] FIRING: [6x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [06:53:43] FIRING: [22x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [07:03:43] RESOLVED: [22x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [07:58:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [08:03:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [10:08:43] FIRING: [4x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [10:13:43] FIRING: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [10:23:43] RESOLVED: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [12:30:37] 06Traffic, 06SRE: Meta query about why we map 31.13.103.0/24 to US - https://phabricator.wikimedia.org/T409735#11362769 (10cmooney) Thanks @ssingh. I'm just reading about this RFC for the first time, I wonder longer term might it be a goal to automate the ingestion of data from such feeds to update our maps a... [14:21:06] 06Traffic, 06SRE: Meta query about why we map 31.13.103.0/24 to US - https://phabricator.wikimedia.org/T409735#11363159 (10ssingh) >>! In T409735#11362769, @cmooney wrote: > Thanks @ssingh. I'm just reading about this RFC for the first time, I wonder longer term might it be a goal to automate the ingestion of... [15:30:08] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: Nokia OSPF alerts not working - https://phabricator.wikimedia.org/T408378#11363473 (10cmooney) 05Open→03Resolved a:03cmooney >>! In T408378#11351612, @colewhite wrote: > In today's case, the alert criteria wasn't met because... [18:20:35] 06Traffic, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast) - https://phabricator.wikimedia.org/T409860 (10ssingh) 03NEW [18:20:57] 06Traffic, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11364269 (10ssingh) [18:23:27] 06Traffic, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11364275 (10ssingh) Initial role can be `insetup::traffic_nftables`. We will reimage to `hcaptcha::proxy` role later, with Debian... [18:24:54] 06Traffic, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11364278 (10ssingh) Once the VMs are up, we will need to enable BGP for all of them in Netbox and then run `homer`. [19:44:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [19:49:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [20:07:42] 06Traffic: Local coding agents should be allowed to fetch wikipedia / wikidata - https://phabricator.wikimedia.org/T409871#11364635 (10Reedy) https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy would be partially related So would https://wikimediafoundation.org/news/2025/11/10/in... [20:07:58] ^ not sure if there's a better task to tag that against [20:20:35] 06Traffic: Local coding agents should be allowed to fetch wikipedia / wikidata - https://phabricator.wikimedia.org/T409871#11364642 (10taavi) 05Open→03Stalled There's not much that can be done here without the details included in HTTP response body. [20:30:08] Reedy: perfectly fine, thanks for tagging [20:32:32] 06Traffic: Local coding agents should be allowed to fetch wikipedia / wikidata - https://phabricator.wikimedia.org/T409871#11364659 (10Monneyboi) Thanks. I'm aware of the discussion on companies scraping the wiki infra, and the value of Wikipedia, and I think this is an important discussion to have. This is howe...