[00:17:35] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414362 (10cmooney) The issue is the speed at which esams is pulling the file in from codfw at, which happens every time. Tbh it doesn't make much se... [00:38:18] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414384 (10Bewfip) According to https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals, esams and drmrs should not... [00:39:22] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414391 (10Dzahn) more notes (that I did not get to save yet, from earlier) - ATS isn't the cause, it's just slow without it (from curl testing from... [01:17:35] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414454 (10Dzahn) - Ganeti host isnt dropping packets either - roughly 0.8% loss is on the network beyond ganeti1033: the switches, the eqiad↔ulsfo... [01:17:54] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414456 (10Dzahn) How about now, everybody? [01:53:41] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414546 (10Bewfip) Has improved for me. Though I’m curious why those dedicated inter-DC links have a relatively high loss rate. [03:17:55] FIRING: [2x] PybalBackendDown: Pybal backend druid1010:0 is down (druid-public-broker_8082) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [07:17:55] FIRING: [2x] PybalBackendDown: Pybal backend druid1010:0 is down (druid-public-broker_8082) - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DPybalBackendDown [08:17:31] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12415042 (10cmooney) >>! In T440068#12414454, @Dzahn wrote: > - roughly 0.8% loss is on the network beyond ganeti1033: the switches, the eqiad↔ulsfo >... [08:23:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12415061 (10cmooney) @Jclark-ctr moved the uplink to the adjacent rack and everything is now working fine. I'll keep this task open... [08:35:26] 06Traffic, 06MediaWiki-Media-Platform-Team, 06ServiceOps, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Datacenter-Switchover: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896#12415082 (10Gehel) [08:42:15] 06Traffic, 06ServiceOps, 10Wikidata, 06Wikidata Platform Team, and 3 others: Migrate the deployment of WDQS UI from wikikube to dse-k8s - https://phabricator.wikimedia.org/T436224#12415252 (10Gehel) [12:28:11] 06Traffic, 10Maps, 06SRE: Possibility to allow Wikimedia Maps usage on all Wikibase Cloud instances - https://phabricator.wikimedia.org/T429191#12416069 (10Anton.Kokh) @MSantos Could you please check if this could be approved? Thank you! [12:35:34] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Add missing wikikube workers to conftool-data - https://phabricator.wikimedia.org/T420729#12416128 (10JMeybohm) I've added all the nodes to conftool data and pooled all but the `wikikube-worker-exp` ones with a weight of 10. [13:45:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12416304 (10Jclark-ctr) Both switches have been Racked in C8 and D5 [15:07:09] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12416592 (10fnegri) I think this task can be resolved, no references are left (only comment lines): `lang=shell-session... [15:18:39] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416633 (10Dzahn) >>! In T440068#12414362, @cmooney wrote: > The issue is the speed at which esams is pulling the file from codfw, which happens every... [15:28:11] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12416665 (10dr0ptp4kt) The replication factor was restored to 2 during a call. In case we hit a situation like this again, the temporary workaround is now documented... [15:28:29] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12416667 (10dr0ptp4kt) 05Open→03Resolved [15:58:15] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416816 (10cmooney) >>! In T440068#12416633, @Dzahn wrote: > Just checking on this part. This should be about "esams <-> eqiad" link. the currently ac... [16:03:53] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416845 (10Dzahn) 05Open→03Resolved a:03Dzahn Thank you very much. Let me close this out once the change is properly in puppet. [16:04:06] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12416848 (10Dzahn) 05Resolved→03Open [16:48:00] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12417085 (10Dzahn) 05Open→03Resolved [16:48:55] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12417086 (10Dzahn) We might also want to do this for certain other services. (gerrit, gitlab, people?) that offer large downloads to end users and... [17:03:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12417125 (10cmooney) >>! In T418012#12411083, @Jclark-ctr wrote: > Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect... [17:08:45] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12417164 (10Dzahn) Follow-up tests, all with cubic (default), 100 MB / 42 MB files: * people2004 (codfw) -> cp3070 (esams), direct: 24 MB/s... [17:22:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417216 (10Jclark-ctr) @cmooney I moved the existing SCS cables from cloudsw1 to the cloudsw3 switches. I verified the SCS connections, u... [17:30:23] bblack: sukhe: do you know offhand if we've ever looked at the https://docs.trafficserver.apache.org/admin-guide/plugins/slice.en.html ATS slice plugin for upload cp? [17:35:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417289 (10cmooney) Cool thanks. If we can run the following links from the Juniper switches in those racks to the new ones that'd be gre... [18:51:44] 06Traffic, 06Commons: HTTP 429 error on image requests on Commons (non-WMF native apps treated as bot-traffic) - https://phabricator.wikimedia.org/T413570#12417635 (10Nylki) If I am not mistaken, the behaviuor appears to have been improved! I am cautiously optimistic after some tests. Yesterday I tested a typ... [19:19:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12417704 (10Jclark-ctr) These cables have been connected >>! In T440313#12417289, @cmooney wrote: > Cool thanks. If we can run the foll... [19:25:59] cdanis: I haven't looked at it before, maybe v g has?