[00:01:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp3069:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=esams&var-instance=cp3069 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:06:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:09:43] FIRING: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3067&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [00:19:43] RESOLVED: HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3067&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [00:31:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:46:40] FIRING: [16x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:51:40] FIRING: [15x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:06:40] RESOLVED: [8x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [03:35:10] 07HTTPS, 06Traffic, 10MediaWiki-Action-API, 10MediaWiki-REST-API, and 4 others: Proposal: fail explicitly and revoke relevant API keys over plain-text HTTP connection for all Wikimedia APIs - https://phabricator.wikimedia.org/T368344#11439149 (10EMill-WMF) > Personally, I'd rather we put effort into elimin... [03:51:37] 07HTTPS, 06Traffic, 10MediaWiki-Action-API, 10MediaWiki-REST-API, and 4 others: Proposal: fail explicitly and revoke relevant API keys over plain-text HTTP connection for all Wikimedia APIs - https://phabricator.wikimedia.org/T368344#11439150 (10Bawolff) Personally, i don't think this mitigation is worth i... [10:13:37] 06Traffic, 06Data-Platform-SRE (2025.11.07 - 2025.11.28), 07Essential-Work, 13Patch-For-Review: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222#11439767 (10Gehel) a:05BTullis→03Gehel [13:36:02] 10netops, 06Infrastructure-Foundations, 06SRE: Nokia SR-Linux ARP resolution bug on v24.10.x+ - https://phabricator.wikimedia.org/T409178#11440290 (10cmooney) Nokia have come back to say they were able to reproduce the issue, and confirm the cause as well as the fact it is not a problem in the latest SR-Linu... [15:17:24] 06Traffic, 10Math, 10PageViewInfo, 06MediaWiki-Platform-Team (Kanban Board), 07OKR-Work: Fix external calls to AQS in Wikimedia extensions - https://phabricator.wikimedia.org/T411641#11440674 (10Tgr) [15:55:18] 10netops, 06cloud-services-team, 10Horizon, 06Infrastructure-Foundations, 10Striker: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783#11440797 (10LSobanski) p:05Triage→03Low [15:56:02] 10netops, 06cloud-services-team, 10Horizon, 06Infrastructure-Foundations, 10Striker: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783#11440802 (10cmooney) Thanks for the task @taavi. I think the idea makes sense, we probably need to take a close look at what they expose an... [16:44:12] 06Traffic, 06collaboration-services, 10Gerrit: ATS/Gerrit: validate TLS hosts for gerrit (revert workaround that skips validation) - https://phabricator.wikimedia.org/T411904#11441053 (10LSobanski) p:05Triage→03Medium [19:20:47] 06Traffic: ncmonitor should automatically guess domain names when adding ncredir entries - https://phabricator.wikimedia.org/T368692#11441564 (10BCornwall) [19:23:16] 06Traffic: ncmonitor should automatically guess domain names when adding ncredir entries - https://phabricator.wikimedia.org/T368692#11441571 (10BCornwall)