[08:58:47] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12388400 (10ayounsi) [09:01:56] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12388405 (10ayounsi) [09:03:34] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12388409 (10ayounsi) > Thank you for this report. There is currently maintenance being performed on some of our trans-Pacific circuits which may be impacting... [10:33:38] 06Traffic, 10hCaptcha, 06Product Safety and Integrity, 06ServiceOps, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12388742 (10MLechvien-WMF) [11:45:44] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12388907 (10isarantopoulos) @ssingh o/ Just a heads up that @kevinbazira has started generating the batch of audio files that will be used in the TTS experiment that will st... [11:54:12] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12388936 (10isarantopoulos) The total size is much smaller than the initial request since we are only generating audio for the Lead section of each article. @kevinbazira can... [12:47:31] 10netops, 06cloud-services-team, 10Horizon, 06Infrastructure-Foundations, and 2 others: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783#12389087 (10aputhin) [13:23:11] changing the liftwing-studio name to hummingbird (it came yesterday when the old name was being already applied so i didn't want to disrupt the existing process): [13:23:11] https://gerrit.wikimedia.org/r/c/operations/dns/+/1348507 [13:23:11] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1348508 [13:23:11] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1348509 [13:23:29] this is just changing what already exists [13:24:44] happy to review; are you planning to push this out today? (asking) [13:33:05] i think the risk is low and the app is not used so i would be happy to [13:34:01] our Friday policy is not to push any changes to the CDN unless it's urgent, even if they are low-risk like these [13:34:20] is there anything on your end that makes this urgent? if so, we can push it. if not, we prefer not to. [13:35:06] the risk essentially is that changes are usually coupled to other stuff and can spill over [14:06:10] no urgent, can wait [14:06:17] so no big problem [14:06:40] if you want to approve this now i can apply it monday [14:07:53] dpogorzelski: yes I will review shortly and +1 [14:08:09] and thanks, please ping us on Monday for a deploy [14:13:54] will do :) [14:52:41] 06Traffic, 10Liberica: Get liberica ready to be deployed on low-traffic loadbalancers - https://phabricator.wikimedia.org/T439840#12389712 (10ssingh) [15:42:20] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 3 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12390120 (10Krinkle) 05Open→03Resolved [18:31:33] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12390833 (10ssingh) Some updates: After @isarantopoulos' message today and given the backends are active, I checked the cache-control stuff to make sure it would align what... [18:41:57] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12390852 (10dr0ptp4kt) `an-druid1005` and `an-druid1007` are full. Landing here for keyword search. I've flagged this on IM. The netflow retention may or may not... [18:55:52] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12390865 (10dr0ptp4kt) Here are some directory-by-directory views on storage right now to get a feel. ` an-druid1005:/srv/druid/segment-cache$ du -sh * 19G banne... [19:27:40] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install lvs102[1-4] - https://phabricator.wikimedia.org/T440030 (10RobH) 03NEW [19:28:34] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install lvs102[1-4] - https://phabricator.wikimedia.org/T440030#12390962 (10RobH) a:03ssingh @ssingh, I've made some assumptions regarding hostnames and networking. Please review and also provide racking deails. Please update the site.pp file with... [19:45:36] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12391011 (10dr0ptp4kt) 05Resolved→03Open [20:04:38] 07HTTPS, 06Traffic, 06SRE, 06Traffic-Icebox, and 2 others: Support Encrypted Client Hello (ECH) on Wikimedia servers - https://phabricator.wikimedia.org/T205378#12391105 (10ssingh) We have captured the current state of the ECH deployment plan at https://meta.wikimedia.org/wiki/Talk:Community_Wishlist/W519#... [21:35:04] 10netops, 06Infrastructure-Foundations: Update `netflow` retention strategy in Druid (too much data) - https://phabricator.wikimedia.org/T387839#12391657 (10dr0ptp4kt) Cross-linking: {T440033} seemed to play a role in (probably) forcing more data onto surviving nodes, right at the time of the month storage nor... [21:36:53] 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062 (10Reedy) 03NEW [21:37:02] 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391672 (10Reedy) [21:43:58] 06Traffic, 06SRE: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391683 (10Reedy) p:05Triage→03High [21:44:53] 06Traffic, 06SRE: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391686 (10Reedy) [21:45:48] 06Traffic, 06SRE: MW Tarball download issues - https://phabricator.wikimedia.org/T440062#12391691 (10Reedy) [21:47:19] 06Traffic, 06SRE: MW Tarball download issues - https://phabricator.wikimedia.org/T440062#12391697 (10taavi) I can reproduce when hitting both esams and drmrs, but not when hitting eqiad. In case traceroutes are helpful, see P96552. [21:48:31] 06Traffic, 06SRE: MW Tarball download issues - https://phabricator.wikimedia.org/T440062#12391698 (10Reedy) [21:49:33] 06Traffic, 06SRE: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391699 (10Reedy) [21:53:32] 06Traffic, 06SRE: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391704 (10taavi) Can reproduce from the esams bastion directly: `lang=shell-session,lines=5 taavi@bast3007 ~ $ wget "https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz" --2026-10-02... [22:38:42] 06Traffic, 06SRE: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391791 (10Scott_French) Pulling a couple of data points together, this is reproducible internally when fetching directly from `releases.discovery.wmnet` (currently a CNAME to `releases1003.eqiad.wmnet`). Wh... [23:10:27] 06Traffic, 06SRE, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391824 (10Scott_French) To recap, @Dzahn noticed that envoy had a 65s timeout on the RouteAction associated with the local listener. Indeed, in a scenario where the overall response hit... [23:15:14] 06Traffic, 06SRE: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068 (10Reedy) 03NEW [23:15:16] 06Traffic, 06SRE, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391839 (10Dzahn) I set `profile::tlsproxy::envoy::upstream_response_timeout: 300` in Hiera on the releases server role with the change linked above. This reloaded envoy proxy via puppe... [23:16:52] 06Traffic, 06SRE, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391840 (10Dzahn) ` root@releases1003:/home/dzahn# curl -s http://127.0.0.1:9631/stats | grep -E 'upstream_rq_timeout|upstream_rq_completed' | grep -v ': 0$' cluster.local_port_80.exte... [23:17:38] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391841 (10Dzahn) [23:19:14] 06Traffic, 06Collaboration-Services, 06SRE: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12391847 (10Dzahn) [23:19:30] 06Traffic, 06Collaboration-Services, 06SRE: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12391848 (10Reedy) [23:32:18] 06Traffic, 06Collaboration-Services, 06SRE: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391863 (10Dzahn) p:05High→03Medium