[11:10:04] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work, 13Patch-For-Review: Dial test2 traffic to have 50/50 split - https://phabricator.wikimedia.org/T405367#11209539 (10hnowlan) 05Open→03Resolved The 50% change has been merged and is rolling out over the next 30 minutes or so. Please be aware of cache wh... [13:02:09] 06serviceops, 10Ceph, 06collaboration-services, 10envoy, and 2 others: Upgrade swift and ceph frontends to envoy v1.29.12 - https://phabricator.wikimedia.org/T405469 (10MatthewVernon) 03NEW [13:28:05] 06serviceops, 10Ceph, 06collaboration-services, 10envoy, and 2 others: Upgrade swift and ceph frontends to envoy v1.29.12 - https://phabricator.wikimedia.org/T405469#11209972 (10MatthewVernon) [13:37:00] 06serviceops, 10Ceph, 06collaboration-services, 10envoy, and 2 others: Upgrade swift and ceph frontends to envoy v1.29.12 - https://phabricator.wikimedia.org/T405469#11210009 (10MatthewVernon) [13:40:51] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job updatequerypages-deadendpages-s5 failed - https://phabricator.wikimedia.org/T405313#11210021 (10A_smart_kitten) Hey #serviceops, please could you grab the stack trace & error time for this when you get an opportunit... [13:43:23] 06serviceops, 10Ceph, 06collaboration-services, 10envoy, and 2 others: Upgrade swift and ceph frontends to envoy v1.29.12 - https://phabricator.wikimedia.org/T405469#11210027 (10MatthewVernon) 05Open→03Resolved [13:45:13] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job updatequerypages-deadendpages-s5 failed - https://phabricator.wikimedia.org/T405313#11210038 (10Clement_Goubert) ` arbcom_plwiki Deadendpages [QueryPage] Wikimedia\Rdbms\DBQueryError from line 1239... [13:48:30] 06serviceops, 06MW-Interfaces-Team, 07OKR-Work: Dial test2 traffic to have 50/50 split - https://phabricator.wikimedia.org/T405367#11210054 (10HCoplin-WMF) Thanks, @hnowlan ! You're the best <3 [13:53:04] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job updatequerypages-deadendpages-s5 failed - https://phabricator.wikimedia.org/T405313#11210087 (10Zabe) Yeah. The wiki got added to the s5 dblist in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/11850... [13:58:42] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job updatequerypages-deadendpages-s5 failed - https://phabricator.wikimedia.org/T405313#11210131 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert Job restarted as it's a short one and will finish befo... [14:01:19] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job updatequerypages-deadendpages-s5 failed - https://phabricator.wikimedia.org/T405313#11210168 (10A_smart_kitten) Thanks @zabe for confirming re. the new wiki being setup :) (cross-linking to {T391009}) >>! In T4... [14:52:11] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210338 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqi... [14:58:17] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210373 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw - fin... [15:02:04] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210404 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw... [15:02:49] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210408 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw - M... [15:03:18] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210410 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw - f... [15:04:10] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210412 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to codfw... [15:04:53] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210423 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw... [15:04:58] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210424 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to codfw... [15:05:32] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210425 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw -... [15:05:34] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210426 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw -... [15:07:11] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210429 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to c... [15:09:40] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210432 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codf... [15:11:57] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210451 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw - fi... [15:29:52] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11210517 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad... [15:34:10] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11210559 (10Jdforrester-WMF) [16:31:06] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-19 (2025-09-23 to 2025-10-07)), 07OKR-Work: Execute test plan for rest.php rerouting - https://phabricator.wikimedia.org/T405368#11210892 (10aaron) From running https://gist.github.com/AaronSchulz/80e2b0f97ae8ff6caf52cb425f8e431c , I'm getting one new unexpected... [16:32:46] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-19 (2025-09-23 to 2025-10-07)), 07OKR-Work: [SPIKE] Test plan for rest.php routes in REST gateway - https://phabricator.wikimedia.org/T400152#11210900 (10aaron) 05Open→03Resolved Script is basically https://gist.github.com/AaronSchulz/80e2b0f97ae8ff6caf5... [16:38:12] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-19 (2025-09-23 to 2025-10-07)), 07OKR-Work: Execute test plan for rest gateway rerouting for rest.php requests and report findings - https://phabricator.wikimedia.org/T405368#11210956 (10aaron) [17:21:24] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-19 (2025-09-23 to 2025-10-07)), 07OKR-Work: Execute test plan for rest gateway rerouting for rest.php requests and report findings - https://phabricator.wikimedia.org/T405368#11211211 (10aaron) I'm also seeing: ` WARNING: POST https://test2.wikipedia.org/w/rest... [17:24:14] 06serviceops, 06collaboration-services, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11211243 (10Dzahn) p:05Triage→03High This seems like it's treated as High priority. That being said, I am not sure if clinic duty is still supposed to make decisions on priorit... [17:32:24] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-19 (2025-09-23 to 2025-10-07)), 07OKR-Work: Execute test plan for rest gateway rerouting for rest.php requests and report findings - https://phabricator.wikimedia.org/T405368#11211270 (10aaron) [17:46:50] 06serviceops, 06collaboration-services, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11211325 (10MoritzMuehlenhoff) >>! In T403663#11211243, @Dzahn wrote: > That being said, I am not sure if clinic duty is still supposed to make decisions on priority at all. It's n... [17:56:41] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward DC Switchover (September 2025) - https://phabricator.wikimedia.org/T399894#11211367 (10EBlackorby-WMF) [17:57:01] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward DC Switchover (September 2025) - https://phabricator.wikimedia.org/T399894#11211372 (10EBlackorby-WMF) all went well with no issues! thanks all. [19:12:31] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-19 (2025-09-23 to 2025-10-07)), 07OKR-Work: Execute test plan for rest gateway rerouting for rest.php requests and report findings - https://phabricator.wikimedia.org/T405368#11211609 (10aaron) >>! In T405368#11211211, @aaron wrote: > I'm also seeing extra heade...