[03:44:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11873981 (10Papaul) [04:35:48] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11873989 (10Ladsgroup) >>! In T414805#11873042, @Nux wrote: > > There are still loads of broken `MediaWiki:Common.css`. I'm usually... [07:16:50] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11874212 (10ops-monitoring-bot) Draining ganeti4006.ulsfo.wmnet of running VMs [07:25:47] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11874223 (10ops-monitoring-bot) Draining ganeti4006.ulsfo.wmnet of running VMs [08:31:05] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11874323 (10Nux) >>! In T414805#11873989, @Ladsgroup wrote: >>>! In T414805#11873042, @Nux wrote: >> >> There are still loads of bro... [10:35:12] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11874600 (10ops-monitoring-bot) Draining ganeti4008.ulsfo.wmnet of running VMs [10:42:04] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11874618 (10ops-monitoring-bot) Draining ganeti4008.ulsfo.wmnet of running VMs [12:07:02] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11874991 (10neriah) >>! In T414805#11874323, @Nux wrote: > And that is on top of WMF staff already making interface edits harder by f... [12:19:45] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11875078 (10ops-monitoring-bot) VM ncredir4003.ulsfo.wmnet switching disk type to plain [12:23:23] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11875113 (10ops-monitoring-bot) VM ncredir4004.ulsfo.wmnet switching disk type to plain [12:28:20] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11875160 (10A_smart_kitten) >>! In T414805#11873989, @Ladsgroup wrote: >>>! In T414805#11873042, @Nux wrote: >> >> There are still l... [12:28:33] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11875171 (10ops-monitoring-bot) VM prometheus4003.ulsfo.wmnet switching disk type to plain [12:32:53] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11875219 (10ops-monitoring-bot) VM netflow4003.ulsfo.wmnet switching disk type to plain [12:49:47] headsup, as part of the prep work for the ulsfo switch replacement, hcaptcha-proxy4003 will briefly go down [12:50:17] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11875311 (10ops-monitoring-bot) VM hcaptcha-proxy4003.wikimedia.org switching disk type to plain [13:09:04] 06Traffic, 06DC-Ops, 06Infrastructure-Foundations: ulsfo switch work May 2026: Host reimaging - https://phabricator.wikimedia.org/T424686#11875371 (10MoritzMuehlenhoff) ganeti4006 and ganeti4008 are free of VMs and temporarily removed as nodes from the ulsfo Ganeti cluster. They can be freely shutdown, moved... [14:07:26] 06Traffic, 06collaboration-services, 10Gerrit, 10Quibble, and 3 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11875668 (10ABran-WMF) 05Stalled→03Resolved Closing that task as the network error rate is now more under control. We will track... [14:18:06] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11875703 (10daniel) >>! In T414805#11875160, @A_smart_kitten wrote: > One potential difference that comes to mind is that -- potentia... [14:33:42] 06Traffic: HTTP 503 error trying to make any edits on Wikipedia - https://phabricator.wikimedia.org/T423991#11875808 (10Ergur) I was able to make some edits a few days ago, but the problem has arisen again. [14:40:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11875866 (10RobH) >>! In T408892#11873637, @Papaul wrote: > @RobH Remote hands instructions are ready @ https://docs.google.com/document/d/1EW6hxHCQjXPy1PXQWlu... [15:25:20] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11876119 (10A_smart_kitten) >>! In T414805#11875703, @daniel wrote: > APIs are maintained as stable interfaces, their evolution is su... [15:37:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11876203 (10RobH) >>! In T408892#11875866, @RobH wrote: >>>! In T408892#11873637, @Papaul wrote: >> @RobH Remote hands instructions are ready @ https://docs.go... [16:06:25] 06Traffic, 06Product Safety and Integrity, 06Security-Team, 072026-user-javascript-incident, and 4 others: Deduplicate CSP between VCL and MediaWiki - https://phabricator.wikimedia.org/T420604#11876345 (10sbassett) @ssingh, @catrope et al - I think we can likely get this same configuration deployed to Wik... [16:23:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11876468 (10Papaul) Yes I can take care of that. [17:19:57] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, and 6 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11876609 (10Nux) >>! In T414805#11875703, @daniel wrote: >>>! In T414805#11875160, @A_smart_kitten wrote: >> While it may be true tha... [18:07:33] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Anthropic - https://phabricator.wikimedia.org/T424785#11876724 (10CDobbins) 05Open→03In progress p:05Triage→03Medium [18:08:31] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Anthropic - https://phabricator.wikimedia.org/T424785#11876731 (10CDobbins) I just updated our DNS records, @bcampbell. Let me know if there's any unexpected behavior or if I can close the ti... [18:41:05] 06Traffic, 10MediaWiki-File-management, 10MediaWiki-General, 06MW-Interfaces-Team, and 5 others: Allow MediaWiki to make authenticated requests to other instances of MediaWiki - https://phabricator.wikimedia.org/T424768#11876878 (10AGhirelli-WMF) p:05Triage→03High [22:24:26] 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Anthropic - https://phabricator.wikimedia.org/T424785#11877498 (10bcampbell) @CDobbins All looks good on the Anthropic end, I'm seeing the domain as verified now. Thanks for your help, feel f...