[03:54:58] 06Project-Admins: Archive ogv.js tag - https://phabricator.wikimedia.org/T429074#12016913 (10Aklapper) FYI I also declined the tasks https://phabricator.wikimedia.org/maniphest/?ids=305916,327422,277669,262483,265394,265425,308341,262482,262974,262494,284769,263086,299101 [05:08:26] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T429055#12016934 (10ABran-WMF) 05Open→03Resolved p:05Triage→03High a:03jasmine_ scrapers deterred by @jasmine_, thank you for that [05:08:48] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://phabricator.wikimedia.org/T429055#12016938 (10ABran-WMF) [05:44:43] (03PS1) 10Dragoniez: zuul: add Rajveer42 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1301881 [05:50:52] 10Gerrit: Gerrit marks patches as having merge conflicts after their parent is merged - https://phabricator.wikimedia.org/T427696#12016999 (10SD0001) I think I've been noticing this for a longer time. Noticed it with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1222740 today (which was a child of https://ge... [06:27:30] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor: Waiting for the completion of castor-save-workspace-cache sometimes takes almost 4 minutes for core tests - https://phabricator.wikimedia.org/T418974#12017084 (10hashar) [06:27:32] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042#12017087 (10hashar) →14Duplicate... [06:27:49] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042#12017089 (10hashar) The root cause... [06:28:00] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): CI not running - integration-castor06 is overloaded and castor-save-workspace-cache is freezed - https://phabricator.wikimedia.org/T429042#12017092 (10hashar) [07:02:43] 06Release-Engineering-Team, 06Infrastructure-Foundations: Add tox-uv support to the tox-v{3,4} Docker images - https://phabricator.wikimedia.org/T421348#12017157 (10elukey) @hashar thanks a lot! Quick question about these images - is Releng still the poc to maintain them, or should SRE think to have their own... [07:18:00] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12017213 (10ABran-WMF) Thanks @ssingh @BBlack for these answers, using `cache_misc` will greatly simplify... [07:18:25] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/IssueTrackerLinks] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1301887 [07:37:51] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/DateTimeTools] Use composer template [integration/config] - 10https://gerrit.wikimedia.org/r/1301889 [08:16:36] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12017426 (10ABran-WMF) the new host advertisement pattern looks OK from Gitlab's UI: {F88605457} And from SSH's point of view: ` $ git clone git@gitlab-... [08:38:45] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12017523 (10ABran-WMF) I've added a wikitech page to help with the migration: https://wikitech.wikimedia.org/wiki/GitLab/Migration [09:06:35] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12017662 (10Jelto) The migration documentation looks good, thank you! I also verified cloning from `git@gitlab-ssh-replica-a.wikimedia.org` and `git@gitl... [09:24:54] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12017736 (10ABran-WMF) [09:25:24] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12017752 (10ABran-WMF) [11:49:23] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12018231 (10MoritzMuehlenhoff) I opened a ticket at https://anonticket.torproject.org/ and mirrors.wikimedia.org has been removed from the dow... [11:49:48] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12018235 (10MoritzMuehlenhoff) Likewise osbpo and the Ubuntu mirror are moved by now, the last remaining part is the Debian mirror. [11:55:38] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 19 - https://phabricator.wikimedia.org/T426164#12018248 (10Jelto) I updated `gitlab-ce` and `gitlab-runner` to 19.0 and proceed with updating the test hosts [11:56:08] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/DateTimeTools] Use composer template [integration/config] - 10https://gerrit.wikimedia.org/r/1301889 (owner: 10Hslater) [11:56:21] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/IssueTrackerLinks] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1301887 (owner: 10Hslater) [11:57:56] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DateTimeTools] Use composer template [integration/config] - 10https://gerrit.wikimedia.org/r/1301889 (owner: 10Hslater) [11:58:02] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/IssueTrackerLinks] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1301887 (owner: 10Hslater) [12:22:11] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12018388 (10ABran-WMF) [12:40:45] (03open) 10kbach: Draft: Add rule reference [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/13 (https://phabricator.wikimedia.org/T422930) [12:46:35] 10Beta-Cluster-Infrastructure: Beta cluster proposed OAuth consumer seems to be lost - https://phabricator.wikimedia.org/T429165 (10Viktoria_Hillerud_WMSE) 03NEW [12:50:03] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#12018526 (10Andrew) For future reference: it turned out that our osbpo mirror was one of two stores for those packcages, so we have set up a r... [12:52:56] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12018553 (10ABran-WMF) >>! In T425441#12017662, @Jelto wrote: > In [clone.pp](https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/hea... [13:14:17] (03update) 10kbach: Draft: Add rule reference [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/13 (https://phabricator.wikimedia.org/T422930) [13:14:45] (03update) 10kbach: Draft: Add rule reference [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/13 (https://phabricator.wikimedia.org/T422930) [13:22:35] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12018755 (10ABran-WMF) [13:46:13] (03CR) 10Jaime Nuche: [C:03+2] jjb: [wikilambda-catalyst-end-to-end-daily] Add ZUUL_PATCHSET [integration/config] - 10https://gerrit.wikimedia.org/r/1301447 (owner: 10Vaughn Walters) [13:48:39] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end-daily] Add ZUUL_PATCHSET [integration/config] - 10https://gerrit.wikimedia.org/r/1301447 (owner: 10Vaughn Walters) [13:58:13] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Phab Scap template vs actual value out of sync for numerous values - https://phabricator.wikimedia.org/T398460#12019090 (10Aklapper) Synced `maniphest.custom-field-definitions` config from production database in https://gitlab.wikimedia.org/repos/... [14:26:14] 10MediaWiki-Releasing, 05MW-1.46-release, 05Release: Release MediaWiki 1.46.0 - https://phabricator.wikimedia.org/T423255#12019301 (10Reedy) [14:48:33] (03CR) 10Hashar: "@vwalters@wikimedia.org Was it due to the Catalyst script having learned about `ZUUL_PATCHSET`?" [integration/config] - 10https://gerrit.wikimedia.org/r/1301447 (owner: 10Vaughn Walters) [14:48:53] (03CR) 10Hashar: [C:03+2] zuul: add Rajveer42 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1301881 (owner: 10Dragoniez) [14:50:20] (03CR) 10Hashar: [C:03+2] "I have generated the Jenkins job." [integration/config] - 10https://gerrit.wikimedia.org/r/1301473 (https://phabricator.wikimedia.org/T423607) (owner: 10Xqt) [14:50:37] (03Merged) 10jenkins-bot: zuul: add Rajveer42 to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1301881 (owner: 10Dragoniez) [14:51:14] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/DrawioEditor] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1301302 (owner: 10Hslater) [14:52:57] (03Merged) 10jenkins-bot: jjb/zuul: re-enable fasttest-pypy after PyPy was updated to 3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1301473 (https://phabricator.wikimedia.org/T423607) (owner: 10Xqt) [14:53:14] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DrawioEditor] Disable selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/1301302 (owner: 10Hslater) [14:56:17] 10Beta-Cluster-Infrastructure, 06Product Safety and Integrity: en betawiki: I received an email that I am no longer a bureaucrat, and I can no longer make people admin, but my Special:UserGroups still says I'm a bureaucrat - https://phabricator.wikimedia.org/T429212#12019784 (10Chlod) [14:56:24] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 19 - https://phabricator.wikimedia.org/T426164#12019786 (10Jelto) The GitLab test instance was upgraded to version v19.0.2 successfully. @brennen @dancy feel free to test anything on the test in... [14:56:43] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 19 - https://phabricator.wikimedia.org/T426164#12019787 (10Jelto) [15:00:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Preferences: Cannot save any Special:Preferences on any betawiki - https://phabricator.wikimedia.org/T429215#12019832 (10Novem_Linguae) [15:03:29] 10Beta-Cluster-Infrastructure: Editing on Beta Cluster does not work - https://phabricator.wikimedia.org/T429099#12019898 (10taavi) [15:03:36] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Preferences: Cannot save any Special:Preferences on any betawiki - https://phabricator.wikimedia.org/T429215#12019910 (10taavi) →14Duplicate dup:03T429099 [15:03:39] 10Beta-Cluster-Infrastructure, 06Product Safety and Integrity: en betawiki: I received an email that I am no longer a bureaucrat, and I can no longer make people admin, but my Special:UserGroups still says I'm a bureaucrat - https://phabricator.wikimedia.org/T429212#12019912 (10taavi) →14Duplicate dup:0... [15:03:41] 10Beta-Cluster-Infrastructure: Beta cluster proposed OAuth consumer seems to be lost - https://phabricator.wikimedia.org/T429165#12019914 (10taavi) →14Duplicate dup:03T429099 [15:08:56] 10Beta-Cluster-Infrastructure: Editing on Beta Cluster does not work - https://phabricator.wikimedia.org/T429099#12019940 (10Reedy) p:05Triage→03High [15:12:39] 10Scap, 10MediaWiki-Internationalization, 06MediaWiki-Platform-Team, 10MW-on-K8s, 13Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714#12019952 (10Krinkle) a:03Krinkle Making visible here that there's a code review request pending. [15:12:45] 10Scap, 10MediaWiki-Internationalization, 06MediaWiki-Platform-Team, 10MW-on-K8s, 13Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714#12019955 (10Krinkle) 05Resolved→03Open [15:13:07] 06Release-Engineering-Team, 10Scap, 06MediaWiki-Platform-Team, 07Developer Productivity: Consider reducing output in pre-sync phase of scap-backport - https://phabricator.wikimedia.org/T323325#12019956 (10Krinkle) [15:30:34] 10Phabricator, 13Patch-For-Review: Requests for changes of the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T428290#12020064 (10neriah) [15:37:50] 10Beta-Cluster-Infrastructure: Editing on Beta Cluster does not work - https://phabricator.wikimedia.org/T429099#12020115 (10dancy) I'm investigating. [15:56:34] 10Beta-Cluster-Infrastructure: Editing on Beta Cluster does not work - https://phabricator.wikimedia.org/T429099#12020215 (10dancy) I found the `/usr/local/bin/wmf-beta-update-databases.py` script stalled on `deployment-deploy04`. Doing some debugging using `strace` I found it blocked on a `SELECT MASTER_GTID_W... [15:57:44] hi releng folks! We're trying to set up deployment from our new private fundraising-ml repo. I think we would want to add a read-only user and give them an SSH key that we can use to pull. [15:58:21] I think fundraising analytics has a similar setup - I see there is a bot 'gitlab-puller-token' or the like with guest access to their private repo [15:58:34] Does that sound like the right setup? [15:59:18] sorry, I should have said private fundraising-ml gitlab repo, not gerrit [16:00:28] 10Phabricator: Requests for changes of the automated weekly Phabricator data for Tech News - https://phabricator.wikimedia.org/T428290#12020238 (10Aklapper) Thanks for the patch! [16:00:57] Should I just make a phab with a public key for that request? [16:01:54] I would say basically copy/paste the lines above into phab. [16:02:02] cool, will do! [16:09:30] 06Release-Engineering-Team: Add read-only GitLab use for deployment of fundraising-ml private repo - https://phabricator.wikimedia.org/T429234 (10Ejegg) 03NEW [16:20:43] castor-save-workspace-cache. Sigh. [16:59:31] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12020525 (10Dzahn) Maybe let me also ask Tyler if 98.8 GB `Thcipriani / Flaky Tests` is expected, needed... [17:00:47] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 19 - https://phabricator.wikimedia.org/T426164#12020530 (10brennen) > @brennen @dancy feel free to test anything on the test instance https://gitlab.devtools.wmcloud.org. I did not spot any critic... [17:02:21] 10Beta-Cluster-Infrastructure: Editing on Beta Cluster does not work - https://phabricator.wikimedia.org/T429099#12020532 (10dancy) 05Open→03Resolved a:03dancy Editing is working again. [17:02:51] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12020535 (10thcipriani) >>! In T428903#12020525, @Dzahn wrote: > Maybe let me also ask Tyler if 98.8 GB `... [17:04:29] 10Beta-Cluster-Infrastructure: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12020549 (10dancy) >>! In T428930#12012467, @Zabe wrote: >>>! In T428930#12011975, @dancy wrote: >> @Zabe I saw that you handled T329577 a few years ago and I'm wondering if you can h... [17:05:08] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12020551 (10Dzahn) Oh, interesting. Thanks for the explanation @thcipriani. Yea, I was only looking at ac... [17:25:32] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic: gitlab behind CDN: serve gitlab.wm.o via text-lb instead of dedicated IPs? - https://phabricator.wikimedia.org/T428903#12020643 (10thcipriani) >>! In T428903#12020551, @Dzahn wrote: > Oh, interesting. Thanks for the explanat... [18:24:56] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930#12020871 (10dancy) 05Open→03Resolved a:03dancy [18:25:26] 10Beta-Cluster-Infrastructure: Set up deployment-db16 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T429245 (10dancy) 03NEW [18:25:33] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/Workflows] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1302228 [18:26:49] !log Rebooting deployment-jobrunner05 [18:26:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:02:00] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12020975 (10Dzahn) Thanks for reporting. I confirm there was no more logrotate under the new location. The httpd module always expects them in /var/log/apache2 O... [19:05:42] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12020984 (10Dzahn) I notice now there is `/srv/gerrit/site_path/review_site/logs/logs` (with 2 logs at the end). Did anyone move those around? [19:08:31] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12020987 (10Dzahn) anyways, no problem here right now for the prod server. ` du -h /srv/gerrit/site_path/review_site/logs/ 14G /srv/gerrit/site_path/review_site... [19:22:40] Project mediawiki-core-doxygen build #21135: 04FAILURE in 4 min 36 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21135/ [19:24:56] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 13Patch-For-Review: gitlab workers ulimit nofiles 1073741816 slows down fakeroot - https://phabricator.wikimedia.org/T426827#12020999 (10dancy) @fgiunchedi Let us know how things work now if you remove your workaro... [19:54:26] 06Release-Engineering-Team: Add read-only GitLab use for deployment of fundraising-ml private repo - https://phabricator.wikimedia.org/T429234#12021136 (10Jgreen) [19:59:01] 06Release-Engineering-Team: Add read-only GitLab use for deployment of fundraising-ml private repo - https://phabricator.wikimedia.org/T429234#12021157 (10Jgreen) [20:10:04] 06Release-Engineering-Team: Add read-only GitLab use for deployment of fundraising-ml private repo - https://phabricator.wikimedia.org/T429234#12021225 (10Jgreen) 05Open→03Resolved p:05Triage→03Medium a:03Jgreen I added the frdeploy user's key for read access. The key setup should be done, and once... [20:26:00] !log Added deployment-db16 (T429245) [20:26:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:26:03] T429245: Set up deployment-db16 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T429245 [20:27:06] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Set up deployment-db16 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T429245#12021267 (10dancy) 05Open→03Resolved p:05Triage→03Medium [20:32:32] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-db16 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:32:37] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-db16 in project deployment-prep - https://phabricator.wikimedia.org/T429257 (10wmcs-alerts) 03NEW [20:34:01] Yippee, build fixed! [20:34:01] Project mediawiki-core-doxygen build #21136: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21136/ [20:43:37] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 2 others: rest-gateway: run system tests via helm test - https://phabricator.wikimedia.org/T424825#12021313 (10daniel) Summary of decisions that went into the proposed implementation: ## Goals... [20:44:11] 06Release-Engineering-Team, 06ServiceOps new, 05FY2025-26 KR 5.1, 06MediaWiki-Platform-Team (Kanban Board), and 2 others: rest-gateway: run system tests via helm test - https://phabricator.wikimedia.org/T424825#12021320 (10daniel) [20:47:32] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-db16 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [20:48:50] !log deployment-db11.deployment-prep will be going read-only soon while db15 is being promoted to primary. [20:48:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:52:24] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate Quibble images from bullseye to bookworm - https://phabricator.wikimedia.org/T362705#12021351 (10hashar) @Jdforrester-WMF this is fantastic. Thank you for having dealt with the migration! [20:53:30] !log deployment-db11.deployment-prep going read-only [20:53:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:00:49] !log deployment-db15 promoted to master (T428930) [21:00:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:00:51] T428930: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930 [21:55:55] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12021598 (10bking) [22:30:01] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12021659 (10bking) The change is working, I one-offed `deployment-cirrussearch15.deployment-prep.eqiad1... [22:30:39] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12021660 (10bking) [23:40:05] !log systemctl restart php8.3-fpm on deployment-jobrunner05 to reload beta db configuration (T428930) [23:40:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:40:08] T428930: Set up deployment-db15 with Trixie and wmf-mariadb1011 - https://phabricator.wikimedia.org/T428930 [23:46:32] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-Rdbms: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910#12021736 (10dancy) 05Open→03Resolved a:03dancy All `deployment-db*` servers referenced by operations/mediawiki-config are running MariaDB 10....