[00:23:13] eileen: do you still need help? [00:27:56] no thank you bd808 [00:28:22] excellent. I will end my day then :) [00:47:37] (03update) 10bd808: Update install script to use run.php [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/75 (https://phabricator.wikimedia.org/T392421) (owner: 10mvolz) [01:54:06] cross-posting: our bullseye base image has been rebuilt this evening to no longer include bullseye-backports in `sources.list`. bullseye-backports appears to have been archived earlier today, which results in issues like those reported in [0] in dependent images (e.g., failure of `apt-get update`). [01:54:06] if you see this, it indicates the dependent image needs rebuilt in order to pick up the new base image. more details and current status in [1]. [01:54:06] [0] https://phabricator.wikimedia.org/T383557#11026050 [01:54:06] [1] https://phabricator.wikimedia.org/T383557 [06:43:17] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Create test Gerrit instance - https://phabricator.wikimedia.org/T363196#11026632 (10ABran-WMF) [08:20:47] (03approved) 10jnuche: rbac: Remove duplicate "pod" grant from nodepool cluster role [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/39 (owner: 10bd808) [08:49:05] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11026855 (10Jelto) [09:00:41] (03merge) 10arnaudb: Preparing gitlab replica switchover [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/74 (https://phabricator.wikimedia.org/T400121) [09:13:07] hey! we'll apply https://phabricator.wikimedia.org/P79713 on gitlab-replica-a and b to switch them over for https://phabricator.wikimedia.org/T400121 [09:15:57] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11026948 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org) started [09:35:37] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11027048 (10Jelto) [09:59:51] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap (SpiderPig 🕸️), 07Essential-Work: SpiderPig should support train deployments - https://phabricator.wikimedia.org/T392610#11027111 (1020after4) SpiderPig is hilarious and awesome. Great job everyone! [10:06:59] 10Beta-Cluster-Infrastructure: api.php on several Beta cluster wikis returns Service Temporarily Unavailable - https://phabricator.wikimedia.org/T400245 (10Lucas_Werkmeister_WMDE) 03NEW [10:09:19] 10Beta-Cluster-Infrastructure: api.php on several Beta cluster wikis returns Service Temporarily Unavailable - https://phabricator.wikimedia.org/T400245#11027154 (10Lucas_Werkmeister_WMDE) So far I can’t see any relevant-looking errors in [Beta Logstash MediaWiki errors](https://beta-logs.wmcloud.org/app/dashboa... [10:14:14] 10Beta-Cluster-Infrastructure: api.php on several Beta cluster wikis returns Service Temporarily Unavailable - https://phabricator.wikimedia.org/T400245#11027186 (10Lucas_Werkmeister_WMDE) Hm, the journal on deployment-mediawiki13.deployment-prep.eqiad1.wikimedia.cloud smells like the API php-fpm pool is totally... [10:23:47] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11027208 (10Jelto) [10:26:39] 10Beta-Cluster-Infrastructure, 06SRE, 07Puppet: Puppet configures kernel.core_pattern |/usr/lib/systemd/systemd-coredump, but systemd-coredump is not installed - https://phabricator.wikimedia.org/T400247 (10Lucas_Werkmeister_WMDE) 03NEW [10:26:45] 10Beta-Cluster-Infrastructure: api.php on several Beta cluster wikis returns Service Temporarily Unavailable - https://phabricator.wikimedia.org/T400245#11027236 (10Lucas_Werkmeister_WMDE) >>! In T400245#11027186, @Lucas_Werkmeister_WMDE wrote: > (No idea why the `|/usr/lib/systemd/systemd-coredump` pattern is s... [10:27:58] 10GitLab, 06collaboration-services, 13Patch-For-Review: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11027242 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.failover (Failover of gitlab from gitlab1004.wikimedia.org to gitlab1003.wikimedia.org) finished [10:29:12] done! [10:31:15] 10GitLab, 06collaboration-services: Gitlab-replica switchover (gitlab1004 → gitlab1003) - https://phabricator.wikimedia.org/T400121#11027262 (10ABran-WMF) 05In progress→03Resolved [10:55:36] 10Beta-Cluster-Infrastructure: api.php on several Beta cluster wikis returns Service Temporarily Unavailable - https://phabricator.wikimedia.org/T400245#11027296 (10Lucas_Werkmeister_WMDE) >>! In T400245#11027232, @Lucas_Werkmeister_WMDE wrote: > without the core dump I think it’s going to be tricky to figure ou... [11:47:25] 10GitLab (Infrastructure), 06collaboration-services: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252 (10Jelto) 03NEW [12:07:09] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11027522 (10Jelto) [12:28:00] (03CR) 10Krinkle: "FYI: I don't (yet) have a fix for Cite tests failing without VisualEditor being installed. Antoine also found a number of other failures w" [integration/config] - 10https://gerrit.wikimedia.org/r/1132543 (https://phabricator.wikimedia.org/T389998) (owner: 10Krinkle) [12:35:07] (03merge) 10jnuche: rbac: Remove duplicate "pod" grant from nodepool cluster role [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/39 (owner: 10bd808) [12:42:57] 10Beta-Cluster-Infrastructure: api.php on several Beta cluster wikis returns Service Temporarily Unavailable - https://phabricator.wikimedia.org/T400245#11027678 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved Since the API came back on its own, and I don’t think there’s much we can do / investigate here, l... [12:50:10] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11027702 (10Jelto) [14:13:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Cloud-VPS: http://169.254.169.254/openstack/latest/user_data semi-regularly unavaliable during Magnum Kubernetes cluster builds - https://phabricator.wikimedia.org/T399596#11028001 (10fnegri) p:05Triage→03Medium [14:30:19] (03open) 10swfrench: make-container-image: remove php-ldap from debug [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/197 [14:44:54] (03update) 10dancy: Train.vue: Fix vscode linter warnings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/908 [14:46:10] (03approved) 10dancy: make-container-image: remove php-ldap from debug [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/197 (owner: 10swfrench) [14:59:43] (03CR) 10LWatson: Zuul: [mediawiki/extensions/ReaderExperiments] add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1170337 (owner: 10Hashar) [15:22:05] (03PS1) 10LWatson: Update ReaderExperiments project to non-production use [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) [15:23:34] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T396372#11028315 (10SBisson) Heads up, we've added a global client side error handler and logger in the Section Translation and Unified da... [15:28:17] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T396373#11028322 (10thcipriani) p:05Triage→03Medium a:03brennen [15:28:53] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T396374#11028328 (10thcipriani) p:05Triage→03Medium a:03hashar [15:29:24] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T396375#11028334 (10thcipriani) p:05Triage→03Medium a:03jeena [15:29:59] (03CR) 10Jforrester: [C:04-1] "Alpha-sort needed. I'll fiddle." [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [15:30:29] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T396376#11028341 (10thcipriani) p:05Triage→03Medium a:03jnuche [15:30:30] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T396373#11028346 (10brennen) [15:31:19] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T396377#11028351 (10thcipriani) p:05Triage→03Medium a:03Aklapper [15:31:35] (03CR) 10LWatson: Update ReaderExperiments project to non-production use (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [15:33:53] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T396378#11028373 (10thcipriani) p:05Triage→03Medium a:03dancy [15:34:22] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.45.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T396379#11028379 (10thcipriani) p:05Triage→03Medium a:03dduvall [15:44:21] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for jrmuizel - https://phabricator.wikimedia.org/T400204#11028437 (10thcipriani) [15:45:18] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for jrmuizel - https://phabricator.wikimedia.org/T400204#11028440 (10thcipriani) Hi @Jrmuizel I found your account, but looks like you signed up for GitLab and Phab yesterday and I can't find any other history. What a... [15:46:05] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen): Gitlab "Review changes" button not working - https://phabricator.wikimedia.org/T400091#11028441 (10thcipriani) [15:47:42] 10Release-Engineering-Team (Radar), 10Wikidata, 10MW-1.45-notes (1.45.0-wmf.11; 2025-07-22): mwext-node20-rundoc jobs failing with `Please configure a github access token` - https://phabricator.wikimedia.org/T400036#11028450 (10thcipriani) [15:48:30] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Support "other assignee" field for tasks with "Task" type - https://phabricator.wikimedia.org/T396905#11028451 (10thcipriani) [15:53:17] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/ReaderExperiments] Move to non-production section [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [15:53:41] (03CR) 10Jforrester: "Tweaked — is the comment correct?" [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [15:54:39] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Meetmendapara - https://phabricator.wikimedia.org/T398769#11028479 (10thcipriani) [15:55:28] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Meetmendapara - https://phabricator.wikimedia.org/T398769#11028489 (10thcipriani) Thanks for linking your MediaWiki account! I can't find much history for your user: what would you like to work on on our GitLab? T... [15:56:29] 10Release-Engineering-Team (Radar), 06MediaWiki-Engineering, 06serviceops, 13Patch-For-Review: Deprecate mwdebugXXXX hosts - https://phabricator.wikimedia.org/T397498#11028494 (10thcipriani) [15:56:51] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap (SpiderPig 🕸️), 10Phatality: SpiderPig log view: Integrate Phatality-style search & reporting features for errors - https://phabricator.wikimedia.org/T397094#11028496 (10thcipriani) [15:56:53] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap (SpiderPig 🕸️): SpiderPig log view: Logstash links should apply expected filters - https://phabricator.wikimedia.org/T397093#11028498 (10thcipriani) [15:56:57] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap (SpiderPig 🕸️): SpiderPig log view: Display temporal info to the user - https://phabricator.wikimedia.org/T397092#11028500 (10thcipriani) [15:57:07] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap (SpiderPig 🕸️), 13Patch-For-Review: SpiderPig log view: Make table sortable by column, default to descending by count - https://phabricator.wikimedia.org/T397091#11028502 (10thcipriani) [16:01:49] 10Gerrit, 10Release-Engineering-Team (Seen), 07Upstream: Investigate Gerrit git repositories having invalid references - https://phabricator.wikimedia.org/T337508#11028523 (10thcipriani) [16:28:39] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap (SpiderPig 🕸️), 07Essential-Work: SpiderPig should support train deployments - https://phabricator.wikimedia.org/T392610#11028638 (10dduvall) >>! In T392610#11027111, @20after4 wrote: > SpiderPig is hilarious and awesome. Great job everyone! Thanks, Mu... [16:38:54] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for jrmuizel - https://phabricator.wikimedia.org/T400204#11028687 (10Jrmuizel) I'd like to make it so that the downloadPageAction in the MinervaNeue skin supports Firefox for Android: https://github.com/wikimedia/medi... [17:02:18] 06Release-Engineering-Team, 10Scap: Migrate code for generating MW K8s images to Scap repository - https://phabricator.wikimedia.org/T318544#11028744 (10thcipriani) 05Open→03Declined >>! In T318544#10138283, @thcipriani wrote: > Updated the task description to note @dancy recently ported all the logic... [17:06:12] (03merge) 10swfrench: make-container-image: remove php-ldap from debug [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/197 [17:11:02] (03CR) 10LWatson: "Thanks for adding the comment. It makes sense here because we're using the extension for prototyping" [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [17:44:55] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Support "other assignee" field for tasks with "Task" type - https://phabricator.wikimedia.org/T396905#11028935 (10brennen) 05Open→03Stalled Also changed: - https://phabricator.wikimedia.org/transactions/editengine/maniphest.task/view/37/ We'... [17:44:57] !log Unblock 91.0.0.0/10 (T400250) [17:44:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:45:40] (03update) 10dancy: Train.vue: Add active train operation to train summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [17:46:38] (03update) 10dancy: Train.vue: Add active train operation to train summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [17:48:28] (03open) 10dancy: Finalize "next" patches when prepping a new train branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/912 (https://phabricator.wikimedia.org/T295925) [17:48:31] (03update) 10dancy: Finalize "next" patches when prepping a new train branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/912 (https://phabricator.wikimedia.org/T295925) [17:48:35] (03approved) 10dduvall: Train.vue: Add active train operation to train summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 (owner: 10dancy) [17:49:26] (03merge) 10dancy: Train.vue: Add active train operation to train summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/905 [17:51:13] (03update) 10dancy: Finalize "next" patches when prepping a new train branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/912 (https://phabricator.wikimedia.org/T295925) [18:00:30] 10Continuous-Integration-Config, 06Abstract Wikipedia team, 10Catalyst, 10Wikifunctions: ${{SCRIPT, template="wikimedia.template"}} is not sending a correct template - https://phabricator.wikimedia.org/T400286#11028975 (10Jdforrester-WMF) [18:04:55] (03PS1) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Use a manual e-mail body, not a template [integration/config] - 10https://gerrit.wikimedia.org/r/1172081 (https://phabricator.wikimedia.org/T400286) [18:06:18] (03CR) 10CI reject: [V:04-1] jjb: [wikilambda-catalyst-end-to-end] Use a manual e-mail body, not a template [integration/config] - 10https://gerrit.wikimedia.org/r/1172081 (https://phabricator.wikimedia.org/T400286) (owner: 10Jforrester) [18:07:54] (03PS2) 10Jforrester: jjb: [wikilambda-catalyst-end-to-end] Use a manual e-mail body, not a template [integration/config] - 10https://gerrit.wikimedia.org/r/1172081 (https://phabricator.wikimedia.org/T400286) [18:09:52] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ReaderExperiments] Move to non-production section [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [18:10:16] (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1172081 (https://phabricator.wikimedia.org/T400286) (owner: 10Jforrester) [18:10:55] 10Continuous-Integration-Config, 10Catalyst, 10Wikifunctions, 10Abstract Wikipedia team (26Q1 (Jul–Sep)), and 2 others: ${{SCRIPT, template="wikimedia.template"}} is not sending a correct template - https://phabricator.wikimedia.org/T400286#11029019 (10Jdforrester-WMF) a:03Jdforrester-WMF [18:11:11] 10Continuous-Integration-Config, 10Catalyst, 10Wikifunctions, 10Abstract Wikipedia team (26Q1 (Jul–Sep)), and 2 others: ${{SCRIPT, template="wikimedia.template"}} is not sending a correct template - https://phabricator.wikimedia.org/T400286#11029021 (10Jdforrester-WMF) 05Open→03In progress A quick fix... [18:11:16] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ReaderExperiments] Move to non-production section [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [18:11:39] (03Merged) 10jenkins-bot: jjb: [wikilambda-catalyst-end-to-end] Use a manual e-mail body, not a template [integration/config] - 10https://gerrit.wikimedia.org/r/1172081 (https://phabricator.wikimedia.org/T400286) (owner: 10Jforrester) [18:11:58] !log Zuul: [mediawiki/extensions/ReaderExperiments] Move to non-production section, for T398483 [18:12:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:12:01] T398483: Image Browsing: Set up a new codebase - https://phabricator.wikimedia.org/T398483 [18:46:03] (03update) 10thcipriani: scap update-next-patches: Add --dry-run flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/910 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [18:48:39] Daimona: You don't happen to remember off-hand why the mediawiki-phan-php* images have ext-imagick installed? That's the only thing they have that's not in the Quibble images, and I'd really love to drop them and just re-use Quibble. [18:48:58] I don't think I did that, but lemme check. [18:49:47] (03merge) 10thcipriani: scap update-next-patches: Add --dry-run flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/910 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [18:50:44] (03update) 10thcipriani: Finalize "next" patches when prepping a new train branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/912 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [18:52:13] That comes from https://gerrit.wikimedia.org/r/c/integration/config/+/498297 by Lego in 2019. The commit message has no explanation, and I didn't find anything obviously related in phab. [18:52:44] Presumably, something in our code uses that and we installed the extension instead of using stubs. [18:53:01] Yeah, I suppose we just forgot to remove it. [18:53:08] We dropped imagick from CI generally ages ago. [18:53:17] I'll give it a whirl and see what happens. [18:53:20] That could be it [18:54:52] Also, MW core has a stub for imagick (in internal_stubs), so that would suffice when running on core. For extensions, we should either make them read internal stubs from core, or move them to the base phan config. [18:55:06] Yeah. [18:55:30] But ext-imagick has no hits for repos that use phan: https://codesearch.wmcloud.org/things/?q=ext-imagick&files=&excludeFiles=&repos= [19:01:52] (03PS2) 10Jforrester: Docker: [quibble-bullseye] Copy Phan running scripts so we can use these images [integration/config] - 10https://gerrit.wikimedia.org/r/1171235 (https://phabricator.wikimedia.org/T397429) [19:05:15] 10Continuous-Integration-Config, 10Catalyst, 10Wikifunctions, 10Abstract Wikipedia team (26Q1 (Jul–Sep)), 07Essential-Work: ${{SCRIPT, template="wikimedia.template"}} is not sending a correct template - https://phabricator.wikimedia.org/T400286#11029212 (10vaughnwalters) >>! In T400286#11029021, @Jdforre... [19:07:39] (03PS1) 10Jforrester: [WIP] jjb: Switch phan jobs over to using Quibble image [integration/config] - 10https://gerrit.wikimedia.org/r/1172087 (https://phabricator.wikimedia.org/T397429) [19:09:22] Hmmmmmm. [19:09:23] `The repository 'http://mirrors.wikimedia.org/debian bullseye-backports Release' does not have a Release file.` [19:09:26] * James_F sighs. [19:10:49] (03PS1) 10C. Scott Ananian: Use trailing slash for Parsoid autoload directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/1172089 (https://phabricator.wikimedia.org/T195211) [19:10:54] Looks like this is T383557. [19:10:54] T383557: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557 [19:12:10] (03open) 10dancy: gitlab-runner: bump image version to alpine-v18.0.5 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/483 (https://phabricator.wikimedia.org/T400289) [19:12:12] (03update) 10dancy: gitlab-runner: bump image version to alpine-v18.0.5 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/483 (https://phabricator.wikimedia.org/T400289) [19:17:51] James_F: following up here from [0] in order to provide faster turnaround [19:17:52] [0] https://phabricator.wikimedia.org/T383557#11029230 [19:18:36] swfrench-wmf: Right. Unfortunately, rebuilding `ci-bullseye` fails with "The repository 'http://mirrors.wikimedia.org/debian bullseye-backports Release' does not have a Release file.". [19:19:05] that's ... surprising [19:19:14] * swfrench-wmf is looking [19:19:28] Let me see if Docker has helpfully not actually pulled the latest Wikimedia image. [19:19:45] ^ this is what I'm wondering, yeah [19:19:47] 06Project-Admins, 06Security-Team, 07SecTeam-Processed, 07Security: Modify security-related Phabricator projects related to incidents and audits - https://phabricator.wikimedia.org/T398840#11029242 (10sbassett) 05In progress→03Resolved [19:19:59] Aha. [19:19:59] Status: Downloaded newer image for docker-registry.wikimedia.org/bullseye:20250723 [19:20:10] Let's try again. [19:20:12] \o/ [19:20:22] Amazing. [19:20:26] It seems to work. [19:20:32] OK, time to update the world. This'll be messy. [19:21:03] o/ [19:21:10] James_F: there's one gotcha that I was discussing with d.ancy earlier: `dockerfiles/shellcheck` [19:21:34] 14Release-Engineering-Team (Deployment services), 14Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 06Security-Team, 07Security, 07Security-Audits: Wikimedia deployers audit - https://phabricator.wikimedia.org/T237696#11029249 (10sbassett) [19:21:42] (03CR) 10Subramanya Sastry: [C:03+1] Use trailing slash for Parsoid autoload directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/1172089 (https://phabricator.wikimedia.org/T195211) (owner: 10C. Scott Ananian) [19:21:58] `shellcheck` is a direct dependent of `ci-bullseye` [19:22:38] but assumes `bullseye-backports` is available [19:22:42] Ah, fun. [19:23:03] So shellechek is now unbuildable? [19:23:04] would switching to `ci-bookworm` be possible? [19:23:15] 10Phabricator, 07SecTeam-Processed, 07Security, 07Security-Audits: Audit members of acl*security for more than x duration of no activity (Jan 2022) - https://phabricator.wikimedia.org/T299400#11029257 (10sbassett) [19:23:30] There's work underway to move everything to bookworm, but not sure if there's a reason shellcheck hasn't be moved. [19:24:17] (03PS1) 10Jforrester: Docker: [ci-bullseye] Re-build now that bullseye-backports is dead [integration/config] - 10https://gerrit.wikimedia.org/r/1172092 (https://phabricator.wikimedia.org/T383557) [19:24:21] Here we go. [19:24:39] (03CR) 10Umherirrender: [C:03+1] "Code in patchdemo looks similiar - https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/blob/c27ec0398a2fce77f18241d53123" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1172089 (https://phabricator.wikimedia.org/T195211) (owner: 10C. Scott Ananian) [19:24:44] (03PS1) 10Ahmon Dancy: dockerfiles: Refresh ci-bullseye and descendants [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) [19:24:49] 54 images to switch. [19:25:17] Ah, dancy, sorry, happy to stand back and let you deal. :-) [19:26:03] (03CR) 10Jforrester: dockerfiles: Refresh ci-bullseye and descendants (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) (owner: 10Ahmon Dancy) [19:27:14] James_F: Is there a way to exercise shellcheck if we try bumping it to ci-bookworm? [19:27:39] dancy: Yeah, I'm just looking. bookworm has shellcheck 0.9.0-1, not 0.8.0, but I'll give it a spin. [19:27:51] If you back out your changes to the image I'll switch it in parallel. [19:27:59] Will do [19:28:31] thank you both! :) [19:30:40] (03PS2) 10Ahmon Dancy: dockerfiles: Refresh ci-bullseye and descendants [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) [19:31:15] (03CR) 10Ahmon Dancy: dockerfiles: Refresh ci-bullseye and descendants (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) (owner: 10Ahmon Dancy) [19:31:16] dancy: Mind if I sneak the shellcheck/bookworm one in first, as it's small? [19:31:23] yes please [19:31:37] (03PS1) 10Jforrester: Docker: [shellcheck] Migrate to bookworm / v0.9.0-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1172095 (https://phabricator.wikimedia.org/T383337) [19:31:44] (03CR) 10Jforrester: [C:03+2] Docker: [shellcheck] Migrate to bookworm / v0.9.0-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1172095 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [19:33:06] (03Merged) 10jenkins-bot: Docker: [shellcheck] Migrate to bookworm / v0.9.0-1 [integration/config] - 10https://gerrit.wikimedia.org/r/1172095 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [19:33:23] !log Docker: [shellcheck] Migrate to bookworm / v0.9.0-1, for T383337 [19:33:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:33:26] T383337: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337 [19:34:20] (03PS1) 10Jforrester: jjb: [shellcheck] Switch job to new, bookworm-based image [integration/config] - 10https://gerrit.wikimedia.org/r/1172096 (https://phabricator.wikimedia.org/T383337) [19:34:52] (03Abandoned) 10Jforrester: Docker: [ci-bullseye] Re-build now that bullseye-backports is dead [integration/config] - 10https://gerrit.wikimedia.org/r/1172092 (https://phabricator.wikimedia.org/T383557) (owner: 10Jforrester) [19:36:22] Yup, shellcheck-on-bookworm works fine. [19:36:32] Only one repo uses it anyway. [19:36:35] So easy to test. :-) [19:36:50] (03CR) 10Jforrester: [C:03+2] "Deployed, tested, yay." [integration/config] - 10https://gerrit.wikimedia.org/r/1172096 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [19:37:17] (03CR) 10Jforrester: [C:03+1] dockerfiles: Refresh ci-bullseye and descendants [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) (owner: 10Ahmon Dancy) [19:37:19] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#11029324 (10Jdforrester-WMF) [19:37:37] yay! [19:37:59] (03PS3) 10Ahmon Dancy: dockerfiles: Refresh ci-bullseye and descendants [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) [19:38:13] (03Merged) 10jenkins-bot: jjb: [shellcheck] Switch job to new, bookworm-based image [integration/config] - 10https://gerrit.wikimedia.org/r/1172096 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [19:39:53] (03CR) 10Ahmon Dancy: [C:03+2] dockerfiles: Refresh ci-bullseye and descendants [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) (owner: 10Ahmon Dancy) [19:41:17] (03Merged) 10jenkins-bot: dockerfiles: Refresh ci-bullseye and descendants [integration/config] - 10https://gerrit.wikimedia.org/r/1172093 (https://phabricator.wikimedia.org/T383557) (owner: 10Ahmon Dancy) [19:42:47] !log Rebuilding ci-buster image and descendants (T383557) [19:42:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:42:49] T383557: Deprecate use of bullseye-backports - https://phabricator.wikimedia.org/T383557 [19:43:41] * dancy crosses fingers [19:44:10] Good luck. [19:45:50] !log Unblock 91.64.0.0/14 (T400234) [19:45:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:50:27] (03CR) 10Eric Gardner: "I think there may be some confusion here." [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [19:51:22] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/ReaderExperiments] Move back to production section [integration/config] - 10https://gerrit.wikimedia.org/r/1172104 [19:52:16] (03open) 10dancy: .gitlab-ci.yml: Use bookworm instead of python3-devel image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/484 (https://phabricator.wikimedia.org/T383557) [19:52:17] (03update) 10dancy: .gitlab-ci.yml: Use bookworm instead of python3-devel image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/484 (https://phabricator.wikimedia.org/T383557) [19:54:13] (03merge) 10dancy: .gitlab-ci.yml: Use bookworm instead of python3-devel image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/484 (https://phabricator.wikimedia.org/T383557) [19:54:15] (03CR) 10Eric Gardner: [C:03+1] "Thanks James – I support this." [integration/config] - 10https://gerrit.wikimedia.org/r/1172104 (owner: 10Jforrester) [19:54:51] (03update) 10dancy: gitlab-runner: bump image version to alpine-v18.0.5 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/483 (https://phabricator.wikimedia.org/T400289) [19:55:47] (03CR) 10Jforrester: "The standard CI package runs `npm test` for you already. Please consider using this entry point ahead of a secondary CI job. https://www.m" [integration/config] - 10https://gerrit.wikimedia.org/r/1172104 (owner: 10Jforrester) [19:55:50] (03open) 10thcipriani: Remove debian-backports from source files [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/150 [19:55:54] (03update) 10thcipriani: Remove debian-backports from source files [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/150 [19:55:56] (03merge) 10dancy: gitlab-runner: bump image version to alpine-v18.0.5 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/483 (https://phabricator.wikimedia.org/T400289) [19:57:29] (03update) 10thcipriani: Remove debian-backports from source files [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/150 [19:58:13] (03CR) 10Eric Gardner: [C:03+1] "Ah ok, cool. I wasn't sure if we needed to manually add some YAML lines to run `node20-browser` or something like that." [integration/config] - 10https://gerrit.wikimedia.org/r/1172104 (owner: 10Jforrester) [19:58:47] !log Upgrading staging gitlab-cloud-runners to 18.0.5 [19:58:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:00:38] !log Upgrading staging gitlab-cloud-runners to 18.0.5 cancelled. Still on 18.0.3 [20:00:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:02:26] (03open) 10dancy: Revert "gitlab-runner: bump image version to alpine-v18.0.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/485 [20:02:29] (03update) 10dancy: Revert "gitlab-runner: bump image version to alpine-v18.0.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/485 [20:03:40] (03merge) 10dancy: Revert "gitlab-runner: bump image version to alpine-v18.0.5" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/485 [20:06:20] (03open) 10dancy: gitlab-runner: bump image version to alpine-v18.0.4 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/486 (https://phabricator.wikimedia.org/T400289) [20:06:23] (03update) 10dancy: gitlab-runner: bump image version to alpine-v18.0.4 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/486 (https://phabricator.wikimedia.org/T400289) [20:07:00] (03merge) 10dancy: gitlab-runner: bump image version to alpine-v18.0.4 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/486 (https://phabricator.wikimedia.org/T400289) [20:10:08] !log Upgrading staging gitlab-cloud-runners to 18.0.4 (T400289) [20:10:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:13:46] (03open) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:13:51] (03update) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:14:55] (03update) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:14:57] (03update) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:14:59] (03update) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:16:29] (03update) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:16:31] (03update) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:16:33] (03update) 10dduvall: ci: Modernize Kokkuri usage and blubber.yaml [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/135 [20:16:38] !log Upgrading production gitlab-cloud-runners to 18.0.4 (T400289) [20:16:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:19:23] James_F: Good news. Images are.... still building [20:29:19] still building is still strictly better than already failing, I guess? [20:29:25] right! [20:29:27] :) [20:30:42] (03merge) 10dancy: Remove debian-backports from source files [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/150 (owner: 10thcipriani) [20:50:58] Images built! [20:54:53] (03PS1) 10Ahmon Dancy: Update all images references to the latest [integration/config] - 10https://gerrit.wikimedia.org/r/1172111 (https://phabricator.wikimedia.org/T383557) [20:57:28] James_F: https://gerrit.wikimedia.org/r/c/integration/config/+/1172111 is ready for your review. [21:11:18] \o/ [21:19:43] (03open) 10volker-e: releases: Bump Codex to 2.2.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/84 [21:23:10] (03PS2) 10C. Scott Ananian: Use trailing slash for Parsoid autoload directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/1172089 (https://phabricator.wikimedia.org/T195211) [21:34:14] (03CR) 10C. Scott Ananian: "https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/184" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1172089 (https://phabricator.wikimedia.org/T195211) (owner: 10C. Scott Ananian) [21:39:28] (03approved) 10dduvall: Train.vue: Fix vscode linter warnings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/908 (owner: 10dancy) [21:56:26] (03CR) 10LWatson: "Sorry for the misunderstanding" [integration/config] - 10https://gerrit.wikimedia.org/r/1172057 (https://phabricator.wikimedia.org/T398483) (owner: 10LWatson) [22:20:38] (03update) 10dancy: Train.vue: Fix vscode linter warnings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/908 [22:22:40] (03merge) 10dancy: Train.vue: Fix vscode linter warnings [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/908 [22:32:36] (03approved) 10thcipriani: Finalize "next" patches when prepping a new train branch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/912 (https://phabricator.wikimedia.org/T295925) (owner: 10dancy) [22:49:31] (03approved) 10catrope: releases: Bump Codex to 2.2.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/84 (owner: 10volker-e) [22:49:34] (03merge) 10catrope: releases: Bump Codex to 2.2.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/84 (owner: 10volker-e) [23:52:14] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-VPS (Quota-requests): Large quota increase for zuul Cloud VPS project - https://phabricator.wikimedia.org/T400305 (10bd808) 03NEW [23:52:27] 10Continuous-Integration-Infrastructure (Zuul upgrade): Set up new project for Zuulv3+ pre-merge and non-image-build workloads - https://phabricator.wikimedia.org/T396247#11029710 (10bd808) [23:52:30] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Cloud-VPS (Quota-requests): Large quota increase for zuul Cloud VPS project - https://phabricator.wikimedia.org/T400305#11029709 (10bd808)