[08:43:52] 10netops, 06Infrastructure-Foundations: eqsin-codfw HE packet loss/latency - oct 2026 - https://phabricator.wikimedia.org/T439947#12410152 (10ayounsi) 05Open→03Resolved a:03ayounsi Thanks Willy ! [08:51:32] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12410199 (10Xqt) @ArthurPSmith: With Pywikibot 11.8, I've disabled maxlag for meta queries, paraminfo and help, and enabled a separate maxlag threshold of 30... [09:18:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12410260 (10cmooney) [09:28:59] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12410314 (10Fabfur) [09:53:05] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12410428 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=59bad94b-4992-4c49-bf42-4f76ff78c6e3) set by cmooney@cumin1004 for 2:00:00 on 16 host(s) and their services with reason:... [11:06:17] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12410769 (10Clement_Goubert) @HCoplin-WMF We're only returning the header for now, and not a 403 response, right? [11:36:25] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12410893 (10cmooney) I've upgraded cr1-codfw now too so we can close this. [11:36:34] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#12410897 (10cmooney) 05Open→03Resolved [12:16:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12411083 (10Jclark-ctr) Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect between the cages** Cables are pre ran thou... [12:18:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12411092 (10Jclark-ctr) Additionally cabled lsw1-C1 Adjusting rack positions so scs-c1 is above lsw and mgmt | Network | Device | Port | Con... [12:41:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12411175 (10Jclark-ctr) Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect between the cages** Cables are pre ran... [12:42:33] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12411177 (10Jclark-ctr) A1 was not cabled I have updated netbox for cableid's for all the uplinks except the spine links since they are not setup. [13:39:32] 06Traffic, 06ServiceOps, 10Wikidata, 06Wikidata Platform Team, and 3 others: Migrate the deployment of WDQS UI from wikikube to dse-k8s - https://phabricator.wikimedia.org/T436224#12411464 (10MLechvien-WMF) For triaging purpose, do you know when you roughly plan to do this? @Gehel [13:45:44] 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12411509 (10cmooney) [16:21:34] 06Traffic, 10MediaWiki-REST-API, 10RESTBase, 10RESTBase-Sunsetting, and 4 others: Mark Mathoid endpoints as deprecated - https://phabricator.wikimedia.org/T431372#12412655 (10Clement_Goubert) Confirmed out of band that we are for now only returning the headers and not blocking traffic. We can do that in th... [17:26:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631 (10cmooney) 03NEW p:05Triage→03Medium [17:27:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12412990 (10cmooney) [17:28:55] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12412998 (10cmooney) [19:08:13] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: thumb.wikimedia.org doesn't have a favicon - https://phabricator.wikimedia.org/T436577#12413442 (10Krinkle) I can reproduce this. Viewing an image in a new tab, results in a rapid chain of cascading failures which seems to start from the favicon down.... [19:09:56] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Open image in new tab produces chain of errors (thumb.wikimedia.org lacks a favicon) - https://phabricator.wikimedia.org/T436577#12413475 (10Krinkle) [19:10:34] 06Traffic, 06MediaWiki-Media-Platform-Team, 07Regression: Viewing image in browser tab produces chain of errors (thumb.wikimedia.org lacks a favicon) - https://phabricator.wikimedia.org/T436577#12413476 (10Krinkle) [19:11:12] 06Traffic: Status code 429 response does not give contact information - https://phabricator.wikimedia.org/T440648 (10Phuzion) 03NEW [22:34:27] 06Traffic, 06Collaboration-Services, 06SRE: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414147 (10Dzahn) notes from debugging: - Apache isn't the problem. It sent the whole 99 MB file in 0.23 seconds (about 435 MB/s), so the disk and Apache are fine. (from c... [23:47:14] 06Traffic, 06Collaboration-Services, 06SRE, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414306 (10Bewfip) Tried downloading mediawiki tarball from my home ISP after noticing slow transfer speed with the API when receiving large responses...