[06:25:28] 06Traffic, 10Lift-Wing, 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Host Qwen 3.6-27B as an inference service - https://phabricator.wikimedia.org/T425680#12084059 (10gkyziridis) === Timeout ceiling blocks the graphs-on test on 1013/1014 === Tried to re-deploy qwen36-27b with `ENFORCE_EAGE... [06:50:16] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12084087 (10fgiunchedi) >>! In T411248#12080727, @ayounsi wrote: >>>! In T411248#12076020, @fgiunchedi wrote: >>>>! In T411248#120747... [09:48:05] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Kubernetes: Calico IPv4 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12084768 (10Gehel) [09:52:40] 10netops, 06Traffic, 10Cloud-Services, 06collaboration-services, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12084857 (10Gehel) [09:52:50] 06Traffic, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 2 others: codfw: rack B8 maintenance - https://phabricator.wikimedia.org/T430929#12084861 (10Gehel) [09:54:19] 06Traffic, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Standardize on a single web proxy hostname - https://phabricator.wikimedia.org/T429338#12084892 (10Gehel) [12:56:57] 06Traffic, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Standardize on a single web proxy hostname - https://phabricator.wikimedia.org/T429338#12085597 (10MoritzMuehlenhoff) >>! In T429338#12023739, @Gehel wrote: > We had a similar discussion on T427457. I'm not sure I understand well enough the difference b... [13:08:44] 06Traffic: Add DNS recrods for VMC/BIMI (Verified Mark Certificate) - https://phabricator.wikimedia.org/T431062#12085616 (10ssingh) [13:16:03] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12085652 (10MoritzMuehlenhoff) [13:20:41] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12085664 (10MoritzMuehlenhoff) I reviewed the Puppet classes and everything should be ready for Trixie. How about we create a separate parallel, test node in magru first and when it's working we... [13:39:45] 06Traffic, 13Patch-For-Review: Add DNS recrods for VMC/BIMI (Verified Mark Certificate) - https://phabricator.wikimedia.org/T431062#12085716 (10ssingh) 05Open→03Resolved ` dig _dnsauth.wikimedia.org +short _2oqvnlebbe7cj4kpj93bkoq38wjfl5a.dcv.digicert.com. ` [14:58:27] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12086001 (10cmooney) Looking at T429338 one thing that occurs to me is does it make sense to Anycast to the LVS? What I mean is use... [15:40:24] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12086094 (10cmooney) >>! In T424478#12085664, @MoritzMuehlenhoff wrote: > I reviewed the Puppet classes and everything should be ready for Trixie. How about we create a separate parallel, test no... [15:47:58] 06Traffic, 10hCaptcha, 06Product Safety and Integrity, 10ServiceOps-Mediawiki, 06SRE: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12086110 (10ssingh) [15:48:11] 06Traffic, 10hCaptcha, 06Product Safety and Integrity, 10ServiceOps-Mediawiki, 06SRE: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12086123 (10ssingh) I am taking the liberty to ServiceOps since they own memcached. [15:51:50] 06Traffic, 07affects-translatewiki.net, 13Patch-For-Review: Translatewiki.net unable to fetch updates from gerrit over https - https://phabricator.wikimedia.org/T430613#12086129 (10ssingh) Can someone with a better understanding of this comment if the issue still persists? If yes, we will tackle it again Mon... [17:22:44] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12086301 (10cmooney) FWIW I upgraded cr3-ulsfo as I had to drain it to reset the PIC and it was on known-bad release 23.4R2-S7. [19:11:09] 06Traffic, 06Data-Persistence, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12086443 (10Ladsgroup) [19:11:36] 06Traffic, 06SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334#12086447 (10Ladsgroup) [19:54:41] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 5 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#12086474 (10Nemoralis) [23:28:50] 06Traffic, 06MediaWiki-Media-Platform-Team: Change the webp threshold based on access distribution - https://phabricator.wikimedia.org/T431150 (10Ladsgroup) 03NEW [23:31:07] 06Traffic, 06MediaWiki-Media-Platform-Team: Change the webp threshold based on access distribution - https://phabricator.wikimedia.org/T431150#12086857 (10Ladsgroup)