[00:48:12] (03PS1) 10Mhurd: Add --skip-npm-install to skip the standalone npm install [integration/quibble] - 10https://gerrit.wikimedia.org/r/1299658 (https://phabricator.wikimedia.org/T427922) [01:21:30] 10Phabricator Translations Extension: Update phabricator translations for upstream i18n changes - https://phabricator.wikimedia.org/T412650#12002969 (10Pppery) I intend to do this and related other stuff on my side unblocked by the upgrade sometime tomorrow. [01:22:25] 10Phabricator Translations Extension, 06translatewiki.net: Drop fund and phortune from Phabricator Translations - https://phabricator.wikimedia.org/T428697 (10Pppery) 03NEW [01:22:29] 10Phabricator Translations Extension, 06translatewiki.net: Drop fund and phortune from Phabricator Translations - https://phabricator.wikimedia.org/T428697#12002982 (10Pppery) [03:05:56] (03update) 10phedenskog: Add dashboard with CI time spent by repo/jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/6 [05:15:38] 10Beta-Cluster-Infrastructure, 06Product Safety and Integrity, 06Security-Team: Unblock IPs for Beta Cluster access - https://phabricator.wikimedia.org/T428709 (10HakanIST) 03NEW [05:23:09] (03PS1) 10Vaughn Walters: jjb: [catalyst-daily-AdvancedSearch] Add AdvancedSearch job [integration/config] - 10https://gerrit.wikimedia.org/r/1299812 (https://phabricator.wikimedia.org/T427013) [06:30:50] (03CR) 10Hashar: [C:03+1] Add --skip-npm-install to skip the standalone npm install [integration/quibble] - 10https://gerrit.wikimedia.org/r/1299658 (https://phabricator.wikimedia.org/T427922) (owner: 10Mhurd) [06:59:38] (03update) 10kbach: Update summary rules based on feedback [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/12 (https://phabricator.wikimedia.org/T428150) [07:56:58] 06Release-Engineering-Team (Radar), 10Scap: scap update-patch can abort and leave /srv/patches in a mess - https://phabricator.wikimedia.org/T428316#12003484 (10jnuche) > scap update-patch leaving a mess is the same problem you solved I solved the immediate cause, not the root problem. > The first step o... [08:26:41] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12003543 (10Jelto) [09:02:52] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12003650 (10Jelto) >>! In T425667#12002746, @Dzahn wrote: > httpd has been reconfigured to log to the gerrit site_path/logs It looks like that broke shipping of... [09:38:39] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12003798 (10Jelto) Apache access and error logs are available again after merging the patch (also backfilled). [09:52:25] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12003871 (10Jelto) [10:25:13] 06Project-Admins, 10WikimediaCustomizations: Split WikimediaCustomizations Phabricator project into subprojects - https://phabricator.wikimedia.org/T428727 (10Tgr) 03NEW [10:50:44] 10GitLab (Pipeline Services Migration🐀), 10Wikispeech-Jobrunner, 10Wikispeech-Text-to-Speech: Migrate Wikispeech services to GitLab - https://phabricator.wikimedia.org/T344741#12004076 (10Sebastian_Berlin-WMSE) Yes, please. [12:17:05] 10Continuous-Integration-Config, 07Accessibility, 07Test-Coverage, 07Upstream: Red-green in PHPUnit code coverage are hard to read - https://phabricator.wikimedia.org/T232915#12004471 (10hashar) [12:18:54] (03open) 10taavi: releases: Bump stylelint packages [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/132 [12:19:01] (03update) 10taavi: releases: Bump stylelint packages [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/132 [12:19:04] (03update) 10taavi: releases: Bump stylelint packages [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/132 [12:38:06] 06Project-Admins, 10WikimediaCustomizations: Split WikimediaCustomizations Phabricator project into subprojects - https://phabricator.wikimedia.org/T428727#12004539 (10Jdforrester-WMF) I don't see the value. Can you give examples of sub-components that you think do/will exist that people will care about that a... [12:38:50] (03update) 10jforrester: releases: Bump stylelint packages [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/132 (owner: 10taavi) [12:38:52] (03update) 10jforrester: releases: Bump stylelint packages [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/132 (owner: 10taavi) [12:38:53] (03approved) 10jforrester: releases: Bump stylelint packages [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/132 (owner: 10taavi) [12:38:54] (03merge) 10jforrester: releases: Bump stylelint packages [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/132 (owner: 10taavi) [14:39:03] (03update) 10phedenskog: Add dashboard with CI time spent by repo/jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/6 [15:15:55] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07JavaScript, 07Upstream: Meme audio doesn't stop with preview regenerate - https://phabricator.wikimedia.org/T99759#12005269 (10valerio.bozzolan) a:05Aklapperβ†’03valerio.bozzolan Yep! I hope I've not destroyed anything with this fix:... [15:21:31] 10Phabricator, 06collaboration-services, 10Lingua-Libre: Enable CORS header for Conduit API requests originating from Toolforge and Cloud VPS - https://phabricator.wikimedia.org/T422256#12005303 (10Aklapper) 05Openβ†’03Declined Unfortunately closing this Phabricator task as no further information has b... [15:23:41] (03PS6) 10Jforrester: Docker: Provide quibble-bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1244903 (https://phabricator.wikimedia.org/T362705) [15:23:56] (03CR) 10Jforrester: Docker: Provide quibble-bookworm (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1244903 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [15:39:00] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12005447 (10bking) [15:52:46] hashar: Would appreciate a quick look at ^ before I deploy and see if it breaks CI. :-) SRE are already shipping bookworm in prod a bit, so this is (now unblocked!) something we should do swiftly. [16:00:41] oh man [16:00:48] Dockerfile leading to copy pasting [16:00:49] :) [16:01:25] the things I have in mind is: new python which affects Python but whatever version is being unit tested [16:01:40] "New" Python is 3.11 [16:01:41] Chrome/Firefox updated which would certainly affects qunit/wdio tests somehow [16:01:58] PHP and its extensions which historically tend to have problems [16:02:21] but I guess you can get those build so we at least have a base line to test with [16:02:26] Yeah. [16:02:30] oh [16:02:40] and given prod targets bullseye/php83 [16:02:48] No. [16:02:50] we could get rid of any bullseye image that is not php83 [16:02:52] Prod is bookworm/php83 [16:02:56] That's the whole point. [16:02:57] yeah [16:03:04] oh so we are lagging out [16:03:06] Or rather, a mixed set. [16:03:12] Yeah. :-( [16:03:13] * hashar nods [16:03:29] but as we introduce new bookworm/php84/85 [16:03:38] maybe we can get rid of the bullseye php84/85 images [16:03:54] Yeah, I was going to migrate everything to bookworm, even php82. [16:04:00] : [16:04:01] ! [16:04:01] But only if the tests all pass. [16:04:08] well I guess go ahead [16:04:12] Maintaining bullseye when we don't need it is bad. [16:04:25] if we end up with a single base OS in short time, it is a net win [16:04:30] If. :-) [16:04:45] (03CR) 10Hashar: [C:03+1] "What we said, go for it so we have a baseline to play with and migrate to ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/1244903 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [16:04:50] (03CR) 10Jforrester: [C:03+2] Docker: Provide quibble-bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1244903 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [16:04:53] that might deserve some announcement somewhere though [16:04:58] thanks for the copy pasta! [16:05:04] :-) [16:05:10] copy-paste-fix [16:06:31] (03Merged) 10jenkins-bot: Docker: Provide quibble-bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1244903 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [16:06:48] !log Docker: Provide quibble-bookworm, for T362705 [16:06:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:06:51] T362705: Migrate Quibble images from bullseye to bookworm - https://phabricator.wikimedia.org/T362705 [16:06:56] tentatively the php extensions are all fine [16:07:24] Yeah, that was the original blocker. [16:07:39] But the last one was opensearch(!) for WMDE's end-to-end testing. [16:07:57] if we got the Jenkins jobs build we can try them [16:08:11] and by hacking in ./utils/zuul-mw-jobs-runner.py , we can run the combinations of jobs against all repos and ssee what happens [16:08:24] though the script is quite terrible :\ [16:09:41] Or just push a single jjb for e.g. php82 and manually test the scary repos? :-) [16:09:52] (Which is my normal way.) [16:17:21] that'll work [16:17:43] what I do is I first run the job manually and filing the various ZUUL_* fields [16:17:48] "2026-06-10 16:14:22 [docker-pkg-build] INFO - Could not find path {path}/test.sh {image}, skipping verification of docker-registry.discovery.wmnet/releng/quibble-bookworm-php82 (image.py:241)" <-- Maybe we should start writing test.sh test scripts? [16:18:08] Yeah. [16:18:11] ZUUL_URL=https://gerrit.wikimedia.org/r [16:18:11] ZUUL_PROJECT=mediawiki/core [16:18:11] ZUUL_BRANCH=master [16:18:11] ZUUL_REF=master [16:18:11] Press build :) [16:24:07] (03PS1) 10Jforrester: jjb: Switch Quibble jobs from bullseye to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300196 (https://phabricator.wikimedia.org/T362705) [16:26:25] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate Quibble images from bullseye to bookworm - https://phabricator.wikimedia.org/T362705#12005649 (10Jdforrester-WMF) 05Openβ†’03In progress a:03Jdforrester-WMF [16:30:15] https://integration.wikimedia.org/ci/job/api-testing-mysql-php83/39/console passes, yay. [16:37:42] OK, pressing the big deploy button. [16:42:15] (03PS2) 10Jforrester: jjb: Switch Quibble jobs from bullseye to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300196 (https://phabricator.wikimedia.org/T362705) [16:42:22] (03CR) 10Jforrester: [C:03+2] jjb: Switch Quibble jobs from bullseye to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300196 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [16:44:16] (03Merged) 10jenkins-bot: jjb: Switch Quibble jobs from bullseye to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300196 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [16:50:13] (03PS1) 10Jforrester: Docker: [quibble] Migrate coverage and fresnel to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300206 (https://phabricator.wikimedia.org/T362705) [16:50:16] (03PS1) 10Jforrester: jjb: Migrate coverage and fresnel to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300207 (https://phabricator.wikimedia.org/T362705) [16:50:36] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: wmse - https://phabricator.wikimedia.org/T427814#12005820 (10brennen) 05Openβ†’03Resolved a:03brennen Created: https://gitlab.wikimedia.org/repos/wmse Invited listed users as owners. (Apologies for the delay; m... [16:50:41] (03CR) 10Jforrester: [C:03+2] Docker: [quibble] Migrate coverage and fresnel to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300206 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [16:52:30] (03Merged) 10jenkins-bot: Docker: [quibble] Migrate coverage and fresnel to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300206 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [16:56:25] 10Continuous-Integration-Infrastructure, 10MediaWiki-libs-Rdbms: Raise Quibble jobs' tested version of MariaDB to 10.6 - https://phabricator.wikimedia.org/T366646#12005875 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF As of just now (via T362705), main CI jobs are using MariaDB 10.11.14 (... [17:14:46] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12005958 (10hashar) This came up during our team meeting. For the **remote rewriting** people can be instructed to use: ` name=$HOME/.gitconfig,lang=ini... [17:16:02] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12005973 (10ABran-WMF) >>! In T425441#12005958, @hashar wrote: > This came up during our team meeting. > > For the **remote rewriting** people can be in... [17:21:12] 10MediaWiki-Releasing, 05MW-1.46-notes, 05MW-1.46-release: Release MW 1.46.0-rc.0 - https://phabricator.wikimedia.org/T423267#12006000 (10matmarex) All done: https://www.mediawiki.org/keys/#:~:text=Bartosz%20Dziewo%C5%84ski https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/refs/tags/1.46.0-rc.0... [17:33:51] (03PS2) 10Jforrester: jjb: Migrate coverage and fresnel to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300207 (https://phabricator.wikimedia.org/T362705) [17:35:12] (03CR) 10Jforrester: [C:03+2] jjb: Migrate coverage and fresnel to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300207 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [17:36:38] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate Quibble images from bullseye to bookworm - https://phabricator.wikimedia.org/T362705#12006120 (10Jdforrester-WMF) 05In progressβ†’03Resolved [17:36:51] (03Merged) 10jenkins-bot: jjb: Migrate coverage and fresnel to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1300207 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [17:38:43] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#12006144 (10Jdforrester-WMF) [17:38:43] (03PS1) 10Jforrester: Docker: Drop quibble-bullseye, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1300216 (https://phabricator.wikimedia.org/T362705) [17:40:18] (03CR) 10CI reject: [V:04-1] Docker: Drop quibble-bullseye, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1300216 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [17:40:56] 10GitLab (Project and group requests), 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: CSP Research - https://phabricator.wikimedia.org/T420485#12006153 (10thcipriani) [17:41:11] 10GitLab (Project and group requests), 06Release-Engineering-Team, 07Essential-Work: GitLab Private Repository Request for: production access meta data - https://phabricator.wikimedia.org/T411642#12006155 (10thcipriani) [17:43:28] (03CR) 10Jforrester: "(Leaving this around for a few days in case we need to revert.)" [integration/config] - 10https://gerrit.wikimedia.org/r/1300216 (https://phabricator.wikimedia.org/T362705) (owner: 10Jforrester) [17:43:45] (03PS1) 10Aude: Zuul: [mediawiki/extensions/ReadingLists] Add SiteMatrix dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1300220 (https://phabricator.wikimedia.org/T428780) [17:44:15] (03PS2) 10Jforrester: Docker: Drop quibble-bullseye, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1300216 (https://phabricator.wikimedia.org/T362705) [17:44:15] (03PS1) 10Jforrester: tests: Test quibble-bookworm, not quibble-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1300221 [17:44:31] (03CR) 10Jforrester: [C:03+2] tests: Test quibble-bookworm, not quibble-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1300221 (owner: 10Jforrester) [17:47:00] (03Merged) 10jenkins-bot: tests: Test quibble-bookworm, not quibble-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1300221 (owner: 10Jforrester) [21:28:34] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:38:34] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:43:34] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:43:46] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T428819 (10wmcs-alerts) 03NEW [21:53:34] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:53:40] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://phabricator.wikimedia.org/T428822 (10wmcs-alerts) 03NEW [22:00:53] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12007334 (10bking) After merging the above changes, I also had to update the "prefix puppet" (hieradata for a specific role o... [22:01:24] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://phabricator.wikimedia.org/T428822#12007336 (10bking) 05Openβ†’03In progress p:05Triageβ†’03Low a:03bking [22:01:54] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://phabricator.wikimedia.org/T428822#12007341 (10bking) This is expected; see T425585 for more context. [22:13:34] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:13:58] (03merge) 10thcipriani: Add dashboard with CI time spent by repo/jobs [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/6 (owner: 10phedenskog) [22:37:21] I did a gerrit service restart - had to. to fix logging (T425667) [22:37:21] T425667: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667 [22:37:36] the good part - no more logs into the root partition now [22:38:18] 10Phabricator Translations Extension, 13Patch-For-Review: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12007414 (10Pppery) 05Stalledβ†’03Open [22:45:36] 10Gerrit, 06collaboration-services: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12007419 (10Dzahn) Thanks for that fix @Jelto . That was on me for forgetting to also change that. After the further change above which switched the symlink around and some manual act... [22:52:25] 10Gerrit, 06collaboration-services: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12007425 (10Dzahn) 05Openβ†’03Resolved I claim everything that is left under /var/log now is not gerrit related and gerrit logs are not ending up in the root partition anymore....