[00:07:32] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11780600 (10SomeRandomDeveloper) Also https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php83-phpu... [06:42:25] 06Traffic, 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Surge in webrequest sequence-id validation check - https://phabricator.wikimedia.org/T422030#11780870 (10JAllemandou) [06:42:36] 06Traffic, 06Data-Engineering (Q3 FY25/26 January 1st - March 31th): Surge in webrequest validation check - https://phabricator.wikimedia.org/T422030#11780871 (10JAllemandou) [07:04:42] 06Traffic, 06Data-Engineering, 06MW-Interfaces-Team, 07OKR-Work: Log Api-User-Agent header in Turnilo - https://phabricator.wikimedia.org/T373871#11780894 (10daniel) Having this would also allow us to decide whether it makes sense to start using Api-User-Agent as an alternative to the normal User-Agent hea... [07:08:42] 06Traffic, 06SRE: IP Block/Throttling relief request: urbipedia.org - Bot attack mitigated - https://phabricator.wikimedia.org/T421650#11780898 (10MoritzMuehlenhoff) p:05Triage→03Medium [07:48:29] 10netops, 06Infrastructure-Foundations: eqiad: pod EF switches upgrade (2026) - https://phabricator.wikimedia.org/T422107 (10ayounsi) 03NEW p:05Triage→03Low [07:48:52] 10netops, 06Infrastructure-Foundations: eqiad: pod EF switches upgrade (2026) - https://phabricator.wikimedia.org/T422107#11780960 (10ayounsi) [07:48:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE, 07Sustainability (Incident Followup): ssw1-f1-eqiad: Fan Spinning Upgraded - https://phabricator.wikimedia.org/T400783#11780961 (10ayounsi) [07:49:11] 10netops, 06Infrastructure-Foundations: eqiad: pod EF switches upgrade (2026) - https://phabricator.wikimedia.org/T422107#11780962 (10ayounsi) [07:49:12] 10netops, 06Traffic, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11780964 (10ayounsi) [07:49:38] 10netops, 06Traffic, 06Infrastructure-Foundations: 2026 Junos upgrade - https://phabricator.wikimedia.org/T416444#11780970 (10ayounsi) [07:49:40] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11780971 (10ayounsi) [08:06:21] 10netops, 06Infrastructure-Foundations: esams: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416450#11781010 (10ayounsi) a:03ayounsi Scheduling this for April 7th at 12:00 UTC - 2h Pinging @ssingh (#traffic) for visibility. [08:40:36] 10netops, 06Infrastructure-Foundations: esams: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416450#11781122 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7822bc9e-76f4-4f51-943c-ae5d8f2f7739) set by ayounsi@cumin1003 for 0:30:00 on 4 host(s) and their services wi... [08:53:03] 10netops, 06Infrastructure-Foundations: esams: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416450#11781139 (10ayounsi) [09:08:58] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11781179 (10ayounsi) [09:21:53] 10netops, 06Infrastructure-Foundations: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871#11781218 (10ayounsi) a:03Papaul Now that we did the switchover, we could focus more on that upgrade. @papaul let me know if you're ok to take care of it. [09:25:55] 10netops, 10DNS, 06Infrastructure-Foundations, 10netbox: Missing includes in DNS repo from Netbox-generated snippets - https://phabricator.wikimedia.org/T422115 (10Volans) 03NEW [09:27:27] 10netops, 10DNS, 06Infrastructure-Foundations, 10netbox, 13Patch-For-Review: Missing includes in DNS repo from Netbox-generated snippets - https://phabricator.wikimedia.org/T422115#11781242 (10Volans) [09:40:45] 06Traffic, 06ServiceOps new, 10ServiceOps-Services-Oids, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of the secure-api.js... - https://phabricator.wikimedia.org/T421464#11781278 [09:45:33] 10netops, 06Infrastructure-Foundations: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873#11781311 (10cmooney) a:03cmooney [09:45:51] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11781312 (10ABran-WMF) thanks for these @SomeRandomDeveloper @DLynch, I've merged a config update. Please let me... [10:21:54] 06Traffic, 06DBA, 07Wikimedia-production-error: Database servers in cluster30 are overloaded - https://phabricator.wikimedia.org/T422127 (10AlexisJazz) 03NEW [10:31:20] 06Traffic, 06DBA, 07Wikimedia-production-error: Database servers in cluster30 are overloaded - https://phabricator.wikimedia.org/T422127#11781594 (10Peachey88) →14Duplicate dup:03T422130 [11:14:27] 06Traffic, 06ServiceOps new, 10ServiceOps-Services-Oids, 06Product Safety and Integrity (Sprint Forsythia (Mar 23 - Apr 10))), 05WE4.2 Bot detection (WE4.2 hCaptcha editing trial): hCaptcha: Stop using urldownloader for health checks of the secure-api.js... - https://phabricator.wikimedia.org/T421464#11781876 [11:17:16] 06Traffic, 06DBA, 10Wikidata, 07Wikimedia-production-error: Fatal exception of type "Wikibase\DataModel\Services\Lookup\EntityLookupException" - https://phabricator.wikimedia.org/T422140#11781880 (10Lucas_Werkmeister_WMDE) [11:19:50] 06Traffic, 06DBA, 10Wikidata, 07Wikimedia-production-error: Fatal exception of type "Wikibase\DataModel\Services\Lookup\EntityLookupException" - https://phabricator.wikimedia.org/T422140#11781882 (10Lucas_Werkmeister_WMDE) ==== Error ==== * mwversion: 1.46.0-wmf.22 * timestamp: 2026-04-02T10:41:51.734Z *... [11:20:31] 06Traffic, 06DBA, 10Wikidata, 07Wikimedia-production-error: Fatal exception of type "Wikibase\DataModel\Services\Lookup\EntityLookupException" - https://phabricator.wikimedia.org/T422140#11781886 (10Lucas_Werkmeister_WMDE) > Timing coincides with T422130. Coincidence? Per “Database servers in cluster26 ar... [11:20:53] 06Traffic, 06DBA, 10Wikidata, 07Wikimedia-production-error: Fatal exception of type "Wikibase\DataModel\Services\Lookup\EntityLookupException" - https://phabricator.wikimedia.org/T422140#11781888 (10Lucas_Werkmeister_WMDE) →14Duplicate dup:03T422130 [11:54:10] 06Traffic, 06ServiceOps new, 10ServiceOps-Services-Oids, 13Patch-For-Review, and 2 others: hCaptcha: Stop using urldownloader for health checks of the secure-api.js file - https://phabricator.wikimedia.org/T421464#11781983 (10OKryva-WMF) a:03kostajh [12:18:14] 06Traffic, 06collaboration-services, 10Gerrit: ATS/Gerrit: validate TLS hosts for gerrit (revert workaround that skips validation) - https://phabricator.wikimedia.org/T411904#11782102 (10ABran-WMF) 05Open→03Resolved @Dzahn closing that one because we removed [[ https://gerrit.wikimedia.org/r/plugins/... [12:25:28] 10netops, 06Infrastructure-Foundations: Create public vlan on eqiad and codfw pods E/F - https://phabricator.wikimedia.org/T422043#11782147 (10ayounsi) [12:27:44] 10netops, 06Traffic, 10DNS, 06Infrastructure-Foundations, and 3 others: Missing includes in DNS repo from Netbox-generated snippets - https://phabricator.wikimedia.org/T422115#11782158 (10Volans) p:05Triage→03Medium I've merged and release the fix, do you want to keep the task open to implement some fo... [12:37:05] 10netops, 06Infrastructure-Foundations: Create public vlan on eqiad and codfw pods E/F - https://phabricator.wikimedia.org/T422043#11782177 (10ayounsi) My initial thought was to start with E/F only but you're right better plan it fully here, especially the IP allocations. [12:41:42] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11782185 (10ayounsi) [13:23:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#11782358 (10Jclark-ctr) [14:56:48] 10netops, 06Infrastructure-Foundations: mr1-eqiad: move from OSPF to BGP - https://phabricator.wikimedia.org/T421238#11782844 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=65cfdda7-c7c9-47d4-b073-5892d3f0a271) set by pt1979@cumin2002 for 1:00:00 on 2 host(s) and their services with reason... [15:27:59] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: mr1-eqiad: move from OSPF to BGP - https://phabricator.wikimedia.org/T421238#11783004 (10Papaul) 05Open→03Resolved BGP is up and OSPF removed [15:57:15] 10netops, 06Infrastructure-Foundations: Create public vlans in eqiad and codfw - https://phabricator.wikimedia.org/T422043#11783127 (10cmooney) Is it maybe an idea to re-use some of the existing vlans? Like if we assign rack A1 as the public rack for the A/B POD we could add all the hosts to //public1-a-eqiad... [16:37:32] 06Traffic, 06SRE: IP Block/Throttling relief request: urbipedia.org - Bot attack mitigated - https://phabricator.wikimedia.org/T421650#11783388 (10Alberto) Thank you very much for your help! I have correctly implemented the User-Agent in my LocalSettings.php for both MediaWiki core and the QuickInstantCommons... [17:01:50] hello traffic friends - any concerns / conflicts if I merge an operations/dns change in a bit? (cleaning up a CNAME for a service that has been turned down: https://gerrit.wikimedia.org/r/1198584) [17:02:43] brett: ^ [17:03:11] swfrench-wmf: Nope, not at all! [17:03:22] awesome, thanks! [17:07:26] 06Traffic, 07OKR-Work, 13Patch-For-Review, 10Test Kitchen (Test Kitchen (Experiment Platform Sprint 22)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11783655 (10KReid-WMF) [17:48:51] 10netops, 06Infrastructure-Foundations: esams: upgrade routers & switches (2026) - https://phabricator.wikimedia.org/T416450#11783830 (10ssingh) >>! In T416450#11781010, @ayounsi wrote: > Scheduling this for April 7th at 12:00 UTC - 2h > Pinging @ssingh (#traffic) for visibility. > > And doing mr1-esams now t... [17:54:07] 10netops, 06Traffic, 10DNS, 06Infrastructure-Foundations, and 2 others: Missing includes in DNS repo from Netbox-generated snippets - https://phabricator.wikimedia.org/T422115#11783873 (10ssingh) Thanks for fixing it but I agree that we need an alert for this otherwise we will miss this again. [18:06:10] 06Traffic, 06collaboration-services, 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11783908 (10SomeRandomDeveloper) Thanks, unfortunately the errors are still occuring: https://integration.wikime...