[06:31:03] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library, 13Patch-For-Review: Import wikipedialibrary.org into wmf redirection services - https://phabricator.wikimedia.org/T400367#11033642 (10Samwalton9-WMF) >>! In T400367#11032844, @BCornwall wrote: > @Samwalton9-WMF, @jsn.sherman A few thi... [10:00:18] 06Traffic, 10HaproxyKafka, 13Patch-For-Review: Prevent HaproxyKafka from hanging - https://phabricator.wikimedia.org/T400199#11034076 (10Fabfur) [10:16:30] 10Acme-chief: acme-chief: Emit Prometheus metrics - https://phabricator.wikimedia.org/T400459 (10taavi) 03NEW [11:05:14] 06Traffic: New software: ProxyTester - https://phabricator.wikimedia.org/T400244#11034291 (10Fabfur) [11:15:13] 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, 13Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117#11034302 (10Fabfur) 05In progress→03Resolved [11:15:59] 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 10HaproxyKafka, and 2 others: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#11034306 (10Fabfur) 05In progress→03Resolved Closing as we can use the HaproxyKafka component to keep track of related tasks [11:16:18] 06Traffic, 10HaproxyKafka: haproxykafka: feature: Ability to print structured messages to stdout - https://phabricator.wikimedia.org/T372344#11034310 (10Fabfur) [12:49:16] 06Traffic, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: ESI test string is still shipped by CentralNotice - https://phabricator.wikimedia.org/T400472#11034545 (10R4356th) Can SRE please confirm if it can be removed? [14:18:35] 06Traffic, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: ESI test string is still shipped by CentralNotice - https://phabricator.wikimedia.org/T400472#11034815 (10ssingh) Thanks for reporting. At least on the Traffic/CDN side, the changes were removed in T308799#9752983 where we set it to fals... [14:43:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5029:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5029 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [14:43:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp3071 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3071 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [14:44:43] FIRING: [2x] HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp3066 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages [14:48:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [14:49:43] RESOLVED: [2x] HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp3066 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages [14:53:40] FIRING: [4x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [14:54:28] 06Traffic, 10HaproxyKafka, 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11034940 (10Fabfur) Note that this happened again ~2025-07-24 14:43 on cp3071, same host [14:58:40] FIRING: [5x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:03:40] FIRING: [6x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:13:43] RESOLVED: HaproxyKafkaExporterDown: HaproxyKafka on cp3071 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3071 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [15:18:40] FIRING: [5x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:23:40] FIRING: [5x] VarnishHighThreadCount: Varnish's thread count on cp5025:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:43:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp5028:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [15:48:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5028:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:08:40] RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp5028:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5028 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [16:35:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Link errors: ssw1-d1-codfw <-> ssw1-f1-codfw - https://phabricator.wikimedia.org/T400253#11035362 (10Jhancock.wm) i cleaned the fiber port and optic. last error was at 16:20 UTC. ish. Gonna let it sit and see if it goes off again. I'll... [16:41:19] 10netops, 06Infrastructure-Foundations, 06SRE: Inaccurate stats reported by cr2-codfw - https://phabricator.wikimedia.org/T400205#11035388 (10cmooney) JTAC 2025-0725-789575 opened [17:02:59] 10Domains, 06Traffic, 06Moderator-Tools-Team, 06The-Wikipedia-Library, 13Patch-For-Review: Import wikipedialibrary.org into wmf redirection services - https://phabricator.wikimedia.org/T400367#11035418 (10Dzahn) >>! In T400367#11033641, @Samwalton9-WMF wrote: > The Wikipedia Library is a WMF-maintained p... [17:10:09] 10netops, 06Infrastructure-Foundations, 10netbox, 06SRE, 13Patch-For-Review: Decom cookbook: delete virtual interfaces from device - https://phabricator.wikimedia.org/T398412#11035461 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin2002 for host sretest2005.codfw.w... [17:26:58] 10netops, 06Infrastructure-Foundations, 10netbox, 06SRE, 13Patch-For-Review: Decom cookbook: delete virtual interfaces from device - https://phabricator.wikimedia.org/T398412#11035493 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin2002 for host sretest2005.codfw.wmnet... [17:32:27] 10netops, 06Infrastructure-Foundations, 10netbox, 06SRE, 13Patch-For-Review: Decom cookbook: delete virtual interfaces from device - https://phabricator.wikimedia.org/T398412#11035506 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin2002 for host sretest2005.codfw.w... [18:35:29] 10netops, 06Infrastructure-Foundations, 10netbox, 06SRE, 13Patch-For-Review: Decom cookbook: delete virtual interfaces from device - https://phabricator.wikimedia.org/T398412#11035699 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin2002 for host sretest2005.codfw.wmnet... [22:56:38] 06Traffic, 10HaproxyKafka, 10Data-Engineering (Q1 FS25/26 July 1st - September 30th), 13Patch-For-Review: Haproxykafka silently stops sending request data to kafka - https://phabricator.wikimedia.org/T400039#11036230 (10Ahoelzl) [22:57:20] 06Traffic, 06Movement-Insights, 10Data-Engineering (Q1 FS25/26 July 1st - September 30th): NEW BUG REPORT: Investigate rise in May 2025 Reader metrics - https://phabricator.wikimedia.org/T395934#11036256 (10Ahoelzl)