[08:01:11] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12348690 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0b63e0fc-c67d-42da-a806-56ac1e145157) set by ayounsi@cumin1004 for 2:00:00 on 3 host(s) and their services with reas... [08:01:49] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12348691 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b0de275b-4fb3-4a76-b054-678595c0ceaf) set by ayounsi@cumin1004 for 2:00:00 on 13 host(s) and their services with rea... [08:38:33] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984#12348809 (10ayounsi) [09:35:56] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 (10Vgutierrez) 03NEW [09:41:32] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12349174 (10Vgutierrez) 05Open→03In progress p:05Triage→03Medium a:03Vgutierrez [09:55:01] 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: GeoIP mapping experiments - https://phabricator.wikimedia.org/T332024#12349276 (10LSobanski) @CDanis are you still planning to work on this or should it be unassigned? [09:55:25] 06Traffic: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828#12349280 (10Vgutierrez) [10:01:56] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#12349322 (10LSobanski) @FCeratto-WMF is this task still pending a response from #infrastructure-foundations?? [10:10:36] 06Traffic, 10Hiddenparma: Can't retrieve factgrid ip range - https://phabricator.wikimedia.org/T438817#12349376 (10Fabfur) >>! In T438817#12349324, @Aklapper wrote: >> We can't fetch FactGrid list of crawler IPs anymore > > @Fabfur: Where do we do this? Adding a project tag is welcome :) Added some details... [10:14:05] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#12349393 (10Marostegui) If I recall correctly this needs #traffic and as far as I know I think Traffic needs a bit more time as they were restructuring how things go... [10:21:25] 10netops, 06Infrastructure-Foundations, 06SRE: HE Packet Loss Issues - https://phabricator.wikimedia.org/T438836 (10cmooney) 03NEW p:05Triage→03High [10:21:33] 10netops, 06Infrastructure-Foundations, 06SRE: HE Packet Loss Issues - https://phabricator.wikimedia.org/T438836#12349437 (10cmooney) [12:49:03] 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: GeoIP mapping experiments - https://phabricator.wikimedia.org/T332024#12349921 (10ssingh) >>! In T332024#12349276, @LSobanski wrote: > @CDanis are you still planning to work on this or should it be unassigned? @CDobbins has been working o... [13:12:20] 06Traffic, 10observability, 06SRE, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12350029 (10dkertesz) @Vgutierrez brought up an interesting point: after a crash of HAProxy, the new instance will load a stale stats file which will result in weird m... [14:03:38] 06Traffic, 13Patch-For-Review: GeoIP mapping experiments - https://phabricator.wikimedia.org/T332024#12350257 (10LSobanski) [16:30:27] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12351266 (10BBlack) >>! In T425216#12278778, @Krinkle wrote: >>>! In T435283#12248179, @BBlack wrote: >>... [19:17:00] 06Traffic, 06DBA, 06Infrastructure-Foundations: Move orchestrator (dborch) to private ipaddrs + CDN - https://phabricator.wikimedia.org/T317179#12352215 (10ssingh) >>! In T317179#12349393, @Marostegui wrote: > If I recall correctly this needs #traffic and as far as I know I think Traffic needs a bit more tim... [19:18:55] 06Traffic, 07Wikimedia-production-error: Error: 503, Backend fetch failed at Wed, 16 Sep 2026 16:50:41 GMT - https://phabricator.wikimedia.org/T438210#12352232 (10ssingh) Hi @Yann, thanks for reporting. Is this still persisting? And is the issue just on wikisource or other wikis as well? [19:19:34] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12352233 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host ncredir2002.codfw.wmnet with OS trixie [19:59:46] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12352399 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host ncredir2002.codfw.wmnet with OS trixie completed: - ncredir2002 (**WARN**) - Downtimed on Icin... [20:54:56] 06Traffic, 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster haproxy does not support `warn-blocked-traffic-after` keyword - https://phabricator.wikimedia.org/T428052#12352640 (10Southparkfan) 05Open→03Resolved a:03BCornwall Thanks to Brett's work on T436468, this hack is no longer needed. I have...