[13:42:25] 06serviceops: mysql.get_core_dbs() exception raised due to missing section - https://phabricator.wikimedia.org/T404464#11193685 (10jcrespo) minor followup (just affects documentation): https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/switchdc/databases/prepare.py#L326 where it says "x1... [14:55:19] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11194093 (10Scott_French) [15:02:27] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11194148 (10Scott_French) I've just made a another pass over the logs and service health dashboards, and things are still looking good. I've also updated the task description to consoli... [15:41:58] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward DC Switchover (September 2025) - https://phabricator.wikimedia.org/T399894#11194303 (10Trizek-WMF) [16:10:43] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194463 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from codfw to eqiad - finished... [16:16:44] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11194509 (10akosiaris) I still have the same concerns as voiced in T359067#9602091, but I also have to be pragmatic. I don't see us solving the bigger registry probl... [16:17:06] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194510 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from codfw to eqiad - finished with status: SUC... [16:17:58] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194526 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad - finished with statu... [16:19:25] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194542 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad - [DRY-RUN] MediaWiki rea... [16:19:47] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194543 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad - finished with status: S... [16:20:40] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194548 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad - finished with status... [16:21:00] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194553 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from codfw to eqiad - finished with statu... [16:21:12] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194554 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from codfw to eqiad - finished with statu... [16:21:29] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194555 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad - [DRY-RUN] MediaWiki re... [16:21:30] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194556 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad - finished with status:... [16:23:02] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194563 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from codfw to eqiad - finished with s... [16:23:43] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194565 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from codfw to eqiad - finished with stat... [16:24:16] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194567 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from codfw to eqiad - finished with status: SU... [16:36:57] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11194613 (10ops-monitoring-bot) jasmine@cumin1003 - Cookbook cookbooks.sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from codfw to eqiad - finished wi... [19:23:51] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11195469 (10Reedy) [19:57:47] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11195552 (10CDanis) >>! In T404742#11194509, @akosiaris wrote: > I still have the same concerns as voiced in T359067#9602091, but I also have to be pragmatic. I don'... [23:36:32] 06serviceops, 06collaboration-services, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11196043 (10Dzahn) All services owned by collaboration-services have been upgraded to 1.29.12-1. ` sudo cumin 'A:owner-collaboration-services' 'dpkg -l | grep envoyproxy' .. 45...