[00:03:23] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#11812465 (10Papaul) Planing on doing the mr router on April 16th at 10am CT [10:18:25] 06Traffic, 06ServiceOps new, 10ServiceOps-Services-Oids, 10Thumbor, 13Patch-For-Review: Thumbor is using an unmantained HAProxy version - https://phabricator.wikimedia.org/T422926#11813600 (10Clement_Goubert) 05Open→03Resolved Thumbor has been upgraded to haproxy 3.2 on trixie. Resolving. [11:11:38] claime: ^^ nice, thanks <3 [12:08:48] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#11813995 (10Marostegui) @ssingh we'd need support from your team to move dborch1003 behind the CDN - is it possible to allocate some time for this with @FCeratto-WMF... [12:13:25] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 5 others: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877#11814024 (10JMeybohm) >>! In T356877#11808865, @cmooney wrote: > Broadly the patch submitted looked good to me, though I see... [12:58:08] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 5 others: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877#11814131 (10cmooney) >>! In T356877#11814024, @JMeybohm wrote: > We can also run the prometheus bird exporter as a sidecar t... [13:01:21] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 2 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#11814156 (10Ladsgroup) Thanks. I asked around to see if anyone would be willing to take a look. [13:47:42] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#11814358 (10ssingh) >>! In T317179#11813995, @Marostegui wrote: > @ssingh we'd need support from your team to move dborch1003 behind the CDN - is it possible to allo... [16:28:51] 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 4 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11815435 (10ssingh) Thanks for the update @sbassett! Sounds good and let us know when you are ready and we can tak... [17:26:25] 10netops, 06Infrastructure-Foundations, 06SRE, 07Incident Severity 3: Row C traffic outage Nov 11 2025 - https://phabricator.wikimedia.org/T409800#11815922 (10MLechvien-WMF) [17:30:58] 06Traffic, 06Product Safety and Integrity, 07Documentation, 05WE4.2 Bot detection: hcaptcha proxy: update wikitech page - https://phabricator.wikimedia.org/T411131#11815972 (10ssingh) [17:35:17] 06Traffic, 07Incident Severity 3, 07Wikimedia-Incident: Bad ATS config led to large volume of 5xx from RESTBase - https://phabricator.wikimedia.org/T421203#11816031 (10MLechvien-WMF) [17:43:32] 06Traffic: hcaptcha-proxy: create logstash dashboard - https://phabricator.wikimedia.org/T411780#11816117 (10MLechvien-WMF) a:05Raine→03None [17:43:33] 06Traffic: Draft hCaptcha SLOs, document SLIs - https://phabricator.wikimedia.org/T411256#11816119 (10MLechvien-WMF) a:05Raine→03None [17:43:36] 06Traffic, 05WE4.2 Bot detection: hcaptcha extension, proxy: Define the backoff and retry strategies - https://phabricator.wikimedia.org/T411115#11816121 (10MLechvien-WMF) a:05Raine→03None [17:43:37] 06Traffic, 05WE4.2 Bot detection: hcaptcha-proxy health checks should also depool sites if their upstream is unreachable - https://phabricator.wikimedia.org/T411191#11816123 (10MLechvien-WMF) a:05Raine→03None [17:43:39] 06Traffic: Improve hcaptcha-proxy alerting - https://phabricator.wikimedia.org/T411251#11816127 (10MLechvien-WMF) a:05Raine→03None [17:43:40] 06Traffic: hcaptcha-proxy monitoring: Monitor hcaptcha.com (i.e. upstream) status to distinguish "outages that aren't our problem" - https://phabricator.wikimedia.org/T411255#11816125 (10MLechvien-WMF) a:05Raine→03None [17:43:46] 06Traffic, 07Serviceops-easywins: hcaptcha-proxy: consider unit tests, integration tests - https://phabricator.wikimedia.org/T411782#11816129 (10MLechvien-WMF) [17:43:50] 06Traffic: Improve hcaptcha-proxy Grafana dashboard - https://phabricator.wikimedia.org/T411254#11816130 (10MLechvien-WMF) a:05Raine→03None [17:44:50] 06Traffic: hcaptcha proxy: bump connection limits + stress test - https://phabricator.wikimedia.org/T411141#11816134 (10MLechvien-WMF) a:05Raine→03None [20:40:04] 06Traffic, 10DNS, 06SRE: [Update DNS Record Request] - wikimedia.org - https://phabricator.wikimedia.org/T423199 (10JKelsoteel-WMF) 03NEW [20:53:42] 06Traffic, 10DNS, 06SRE: [Update DNS Record Request] - wikimedia.org - https://phabricator.wikimedia.org/T423199#11816927 (10BCornwall) 05Open→03In progress a:03BCornwall [21:44:11] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Miro - https://phabricator.wikimedia.org/T423199#11817142 (10Dzahn) [22:06:06] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Miro - https://phabricator.wikimedia.org/T423199#11817200 (10BCornwall) Hi, @JKelsoteel-WMF ! This has been deployed - I'm going to go ahead and close this; Please do re-open if something... [22:06:12] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Miro - https://phabricator.wikimedia.org/T423199#11817203 (10BCornwall) 05In progress→03Resolved