[00:00:47] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Infrastructure-Foundations, 06SRE: Update default GitLab runner image to a base image without mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423971 (10thcipriani) 03NEW [00:03:07] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Infrastructure-Foundations, 06SRE: Rebuild dev-images using a base image without mirrors.wikimedia.org in the apt sources - https://phabricator.wikimedia.org/T423972 (10thcipriani) 03NEW [01:15:32] I'm just wanting feedback on this phab https://phabricator.wikimedia.org/T423961 [03:36:36] 10Gerrit, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Figure out how to maintain cv repo - given it is primarily committed to by non-WMF - https://phabricator.wikimedia.org/T423961#11842387 (10Peachey88) [03:37:20] eileen: ive tagged it with the gerrit team so it appears on their board [03:37:32] thanks p858snake|cloud [03:37:44] 10Beta-Cluster-Infrastructure, 07Chinese-Sites: 2FA removal and request for sysop and interfaceadmin on zh.wikipedia.beta.wmcloud.org - https://phabricator.wikimedia.org/T423830#11842402 (10Shizhao) [04:13:34] (03PS1) 10Freephile: Add Greg Rundlett (freephile) to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1275571 [04:59:10] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10dev-images, 06Infrastructure-Foundations, 06SRE: Rebuild dev-images using a base image without mirrors.wikimedia.org in the apt sources - https://phabricator.wikimedia.org/T423972#11842523 (10A_smart_kitten) [07:59:36] RECOVERY - gerrit process on gerrit2002 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /srv/gerrit/site_path/review_site/bin/gerrit.war daemon -d /srv/gerrit/site_path/review_site https://wikitech.wikimedia.org/wiki/Gerrit [08:03:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11842797 (10Jelto) The alert mentioned above is resolved after removing the manually created systemd... [08:30:21] 10Phabricator (Upstream), 07Upstream: Duplicate entry exception when quickly adding/removing task token - https://phabricator.wikimedia.org/T363518#11842881 (10Aklapper) Wanted to try again but I fail to reproduce this locally now. I throttle to something like "Regular 2G" in Firefox' Developer Tools, open the... [09:06:15] 10MediaWiki-Releasing, 06translatewiki.net, 06LPL Essential (FY2025-26 Q3), 10LPL Projects (Other), and 2 others: Configure Translatewiki.net for REL1_46 translation backports - https://phabricator.wikimedia.org/T423265#11843092 (10Wangombe) a:03Wangombe [09:52:42] (03PS2) 10Samtar: Add codenamenoreste to allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1271727 [09:57:00] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06Infrastructure-Foundations, 06SRE: Update default GitLab runner image to a base image without mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423971#11843238 (10A_smart_kitten) [10:53:32] PROBLEM - gerrit process on gerrit1003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /var/lib/gerrit/review_site/bin/gerrit.war daemon -d /var/lib/gerrit/review_site https://wikitech.wikimedia.org/wiki/Gerrit [10:59:32] RECOVERY - gerrit process on gerrit1003 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-17-openjdk-amd64/bin/java .*-jar /srv/gerrit/site_path/review_site/bin/gerrit.war daemon -d /srv/gerrit/site_path/review_site https://wikitech.wikimedia.org/wiki/Gerrit [11:14:24] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11843511 (10Jelto) The replica host `gerrit1003` has been migrated to `/srv/gerrit`. Full runbook o... [11:14:42] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11843517 (10Jelto) [11:42:54] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Add structured logger - https://phabricator.wikimedia.org/T424011#11843605 (10jnuche) [11:43:41] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Add structured logger - https://phabricator.wikimedia.org/T424011#11843607 (10jnuche) a:03jnuche [12:59:38] 10Continuous-Integration-Config, 06Traffic: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11844005 (10ssingh) The page is cached for an hour on the CDN because of the `cache-control` header emitted by the application laye... [13:14:07] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Traffic: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11844063 (10A_smart_kitten) [13:24:48] (03CR) 10Jforrester: [C:03+2] "Note that after this config change, WikimediaCustomizations must not fail CI if CentralAuth isn't installed, or you'll break CI for every " [integration/config] - 10https://gerrit.wikimedia.org/r/1275487 (https://phabricator.wikimedia.org/T420548) (owner: 10Aude) [13:26:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add CentralAuth dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1275487 (https://phabricator.wikimedia.org/T420548) (owner: 10Aude) [13:27:43] !log Zuul: [mediawiki/extensions/WikimediaCustomizations] Add CentralAuth dependency, for T420548 [13:27:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:27:45] T420548: Finalize the maintenance import script - https://phabricator.wikimedia.org/T420548 [13:30:22] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1275914 (https://phabricator.wikimedia.org/T415254) [13:30:37] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1275914 (https://phabricator.wikimedia.org/T415254) (owner: 10Jforrester) [13:34:07] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1275914 (https://phabricator.wikimedia.org/T415254) (owner: 10Jforrester) [13:34:31] !log Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep, for T415254 [13:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:34:35] T415254: Migrate "WikiLambda API" instrument to use the Test Kitchen SDK - https://phabricator.wikimedia.org/T415254 [13:39:57] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06collaboration-services, 06Traffic: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11844285 (10LSobanski) [14:12:15] 10Continuous-Integration-Config, 10WikimediaApiPortal, 07ci-test-error: mediawiki-node24 failure for skin WikimediaApiPortal (due to node-sass) - https://phabricator.wikimedia.org/T423954#11844563 (10Jdforrester-WMF) 05Openβ†’03In progress a:03Jdforrester-WMF [14:13:47] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 07Essential-Work: Deploy Phab/Phorge 2026-04-21 - https://phabricator.wikimedia.org/T424033 (10brennen) 03NEW [14:15:51] (03open) 10brennen: Update submodules for 2026-04-21 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/100 (https://phabricator.wikimedia.org/T424033) [14:34:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11844808 (10Dzahn) Together with Tyler we succesfully connected new jenkins to CI yesterday. Issues with authe... [14:44:59] 10Continuous-Integration-Config, 10WikimediaApiPortal, 07ci-test-error, 13Patch-For-Review: mediawiki-node24 failure for skin WikimediaApiPortal (due to node-sass) - https://phabricator.wikimedia.org/T423954#11844901 (10Jdforrester-WMF) 05In progressβ†’03Resolved [15:00:51] (03merge) 10brennen: Update submodules for 2026-04-21 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/100 (https://phabricator.wikimedia.org/T424033) [15:08:00] 10Phabricator (2026-04-21), 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Deploy Phab/Phorge 2026-04-21 - https://phabricator.wikimedia.org/T424033#11845208 (10Aklapper) 05Openβ†’03Resolved a:03brennen This deployment merged: * Extensions: {T419751} in https://gitlab.wikimedia.org/repos/pha... [15:09:08] 10Phabricator (2026-04-21), 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions: Duplicated symbol SetSubtypeHeraldAction - https://phabricator.wikimedia.org/T419751#11845224 (10Aklapper) 05Openβ†’03Resolved This got deployed today. [15:19:30] 10Phabricator, 06Release-Engineering-Team, 07Essential-Work: devtools: Create new Phab/Phorge test instance on Debian Trixie - https://phabricator.wikimedia.org/T424055 (10brennen) 03NEW [15:20:25] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: devtools: Create new Phab/Phorge test instance on Debian Trixie - https://phabricator.wikimedia.org/T424055#11845300 (10brennen) [15:25:26] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: devtools: Create new Phab/Phorge test instance on Debian Trixie - https://phabricator.wikimedia.org/T424055#11845329 (10brennen) [15:28:04] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11845355 (10brennen) [15:28:25] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11845359 (10brennen) [15:31:22] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11845370 (10brennen) a:05brennenβ†’03None Un-cookie-licking this for the moment, let me know... [15:40:41] (03CR) 10Zfilipin: documentation: Use tmpfs locally to mimic production [integration/quibble] - 10https://gerrit.wikimedia.org/r/1271698 (https://phabricator.wikimedia.org/T423127) (owner: 10Phedenskog) [15:40:45] (03CR) 10Zfilipin: [C:03+2] documentation: Use tmpfs locally to mimic production [integration/quibble] - 10https://gerrit.wikimedia.org/r/1271698 (https://phabricator.wikimedia.org/T423127) (owner: 10Phedenskog) [15:40:47] hey andre brennen, heads-up that https://phabricator.wikimedia.org/settings/user/A_smart_kitten/ currently errors for me with "Source file "/srv/deployment/phabricator/deployment-cache/revs/ce0ec304455c61a69a3d35b354c1d1d60e26bda6/translations/src/translations/TranslatewikiCoreUk.php" failed to load. " [15:42:01] urgh. thanks for letting us now. damn [15:42:16] (also have there been some changes in the most recent update to how Phab treats plural-values in strings? i could swear that something like https://phabricator.wikimedia.org/T415723#11844915 previously said "removed a project", but it now says "removed 1 project(s)") [15:42:56] yes there have been changes like that [15:44:40] yepp, I can confirm that I cannot access https://phabricator.wikimedia.org/settings/panel/datetime/ or https://phabricator.wikimedia.org/settings/panel/language/ [15:45:07] I didn't spot any issues merging that translations patch locally though, maybe I should add that to the future testing plan. sigh [15:45:54] 'yes there have been changes like that' -- hmm. seems on its face like a potential step backward in terms of plural support. but i haven't yet found the actual code that got changed re. this, so i can't assess whether e.g. if there was another reason for moving to those types of strings :) [15:46:43] there have been 1 change(s) [15:47:19] we can hop on that meeting again if you need another deploy today [15:48:34] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11845433 (10A_smart_kitten) [15:49:57] probably deploying again is a good idea, but I first need to find out _how_ to fix that :-/ [15:51:53] (re the plural-support thing) speculation: okay so looking at https://github.com/phorgeit/phorge/blob/86656e94362bb5de52241050ecdef17859446f5a/src/infrastructure/internationalization/translation/PhabricatorUSEnglishTranslation.php#L69-L80, seems like the plural-support versions of those strings might be coded as part of the _US english translation_. [15:51:56] so somehow, maybe, after the deployment, my account might now be reading from a different english translation/version than before(?) [15:53:30] also @anyone let me know if you want me to create a task for the error when visiting Phab settings [15:54:59] feel feee to, yes [15:57:34] (03Merged) 10jenkins-bot: documentation: Use tmpfs locally to mimic production [integration/quibble] - 10https://gerrit.wikimedia.org/r/1271698 (https://phabricator.wikimedia.org/T423127) (owner: 10Phedenskog) [16:00:30] ack; T424059 filed [16:00:30] 10Phabricator, 10Phabricator Translations Extension, 07Regression: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059 (10A_smart_kitten) 03NEW [16:00:30] T424059: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059 [16:02:28] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: CAPI: Add structured logger - https://phabricator.wikimedia.org/T424011#11845481 (10jnuche) 05Openβ†’03Resolved [16:02:42] 10Phabricator, 10Phabricator Translations Extension, 07Regression: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059#11845482 (10Aklapper) p:05Triageβ†’03High [16:03:13] (03PS1) 10EMcFarland: GrowthExperiments: Add CentralNotice as a dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1275969 (https://phabricator.wikimedia.org/T422082) [16:04:50] 10Phabricator, 10Phabricator Translations Extension, 07Regression: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059#11845490 (10A_smart_kitten) Side-note that may or may not be relat... [16:06:49] * brennen catches up on scrollback [16:07:34] andre, A_smart_kitten, mutante: sorry, i just walked over to the cafe and logged back on. i can run a fresh deploy as soon as one is needed. [16:08:26] (03CR) 10Jforrester: "I'm a bit confused by this commit message. GrowthExperiments CI is currently fine, and I think this patch is saying that the proposed new " [integration/config] - 10https://gerrit.wikimedia.org/r/1275969 (https://phabricator.wikimedia.org/T422082) (owner: 10EMcFarland) [16:08:34] reverting is always an option. [16:10:36] 10Phabricator, 10Phabricator Translations Extension, 07Regression: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059#11845515 (10Aklapper) I cannot reproduce this issue locally before... [16:12:24] 10Phabricator, 10Phabricator Translations Extension, 13Patch-For-Review, 07Regression: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059#11845524 (10Pppery) Okay, I'm apparently an... [16:14:09] 10Phabricator, 10Phabricator Translations Extension, 13Patch-For-Review, 07Regression: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059#11845525 (10Pppery) And yes, the missing plu... [16:14:43] brennen: we have a contributed fix, and it fixes the issue for me locally. I just merged it ^ [16:14:56] but need to still deploy that, translations repo only [16:15:34] andre: ack. will prep deploy, one moment. [16:16:29] brennen: oh thank you. If you wanna see my damn face again on some video screen, just tell :P [16:18:11] i'm sitting in a crowded coffeeshop so i think i can spare surrounding patrons the me-on-a-video-call experience. mutante, suppose that silence is still in place? [16:20:18] (03open) 10brennen: Update translation submodule for T424059 [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/101 (https://phabricator.wikimedia.org/T424059) [16:20:26] (03merge) 10brennen: Update translation submodule for T424059 [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/101 (https://phabricator.wikimedia.org/T424059) [16:24:23] 10Phabricator (2026-04-21), 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Deploy Phab/Phorge 2026-04-21 - https://phabricator.wikimedia.org/T424033#11845581 (10Aklapper) For the records, this caused a regression in {T424059} that language related personal user settings were inaccessible for abo... [16:24:24] 10Phabricator, 10Phabricator Translations Extension, 06Release-Engineering-Team (Doing 😎), 07Regression: 'Unhandled Exception ("Exception") / Source file "TranslatewikiCoreUk.php" failed to load' when visiting Phabricator settings pages - https://phabricator.wikimedia.org/T424059#11845576 (10Aklapper)... [16:24:37] Works again. Big thanks everyone! <3 [16:24:52] <3 [16:25:24] thanks all. sorry for caffeine-related post-deploy absence. :) [16:25:26] * andre hates these moments but loves these people [16:25:51] all good, not a super bad issue, A_smart_kitten still managed to file a bug :D [16:26:10] * andre goes back to the original plan for today (washing dishes) [16:26:20] brennen: on the contrary, it seemed like you were here at the perfect moment to deploy the fix :p [16:26:31] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work: Add a zuul tenant config on the zuul scheduler host (zuul1001) - https://phabricator.wikimedia.org/T406384#11845593 (10dduvall) 05Openβ†’03Resolved The `zuul-scheduler` is now conf... [16:26:42] note to myself: expand test plan for translations patches next time [16:34:33] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: Ensure zuul-executor can prepare workspaces and dispatch builds - https://phabricator.wikimedia.org/T424061 (10dduvall) 03NEW [16:34:55] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: Ensure zuul-executor can prepare workspaces and dispatch builds - https://phabricator.wikimedia.org/T424061#11845658 (10dduvall) p:05Triageβ†’03Medium [16:35:05] !log Unblock 79.116.0.0/16 [16:35:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:36:40] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Review MSantos access - https://phabricator.wikimedia.org/T424025#11845693 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper According to https://phabricator.wikimedia.org/project/manage/61/ you never were, hmm. Added now after verifying the chain h... [16:36:57] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: Add MSantos to WMF-NDA group in Phabricator - https://phabricator.wikimedia.org/T424025#11845697 (10Aklapper) [16:43:43] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11845787 (10Dzahn) [17:01:26] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846019 (10Dzahn) Here is a doc with "the plan" (tm) how to move... [17:02:03] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846024 (10Dzahn) @Arnoldokoth Could you comment on the quota si... [17:09:37] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846054 (10Arnoldokoth) @Dzahn The quota looks healthy. I think... [17:11:26] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846060 (10Dzahn) [17:11:54] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846062 (10Dzahn) Thank you @Arnoldokoth ! Sounds good. I don't... [17:12:33] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator, 07Essential-Work: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846071 (10Dzahn) [17:28:40] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11846181 (10Dzahn) [17:28:41] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Ensure zuul-executor can prepare workspaces and dispatch builds - https://phabricator.wikimedia.org/T424061#11846180 (10Dzahn) [17:28:54] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: zuul1002 fails to compile in PCC - https://phabricator.wikimedia.org/T421232#11846183 (10Dzahn) [17:28:55] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11846184 (10Dzahn) [17:29:20] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11846189 (10Dzahn) [17:30:39] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11846194 (10Dzahn) Since https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275972 added a... [17:30:46] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: Ensure zuul-executor can prepare workspaces and dispatch builds - https://phabricator.wikimedia.org/T424061#11846195 (10Dzahn) Since https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275972 added a firew... [17:31:10] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1275985 [17:31:16] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11846201 (10Dzahn) Since https://gerrit.wikimedia.org/r/c/operations/puppet/+/1275972 added a firewall rule the executors can now t... [17:31:55] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 13Patch-For-Review: Ensure PipelineLib works on new Jenkins host - https://phabricator.wikimedia.org/T423968#11846208 (10Dzahn) [17:31:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11846209 (10Dzahn) [17:46:01] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T390531 https://phabricator.wikimedia.org/T394858 https://phabricator.wikimedia.org/T396106) [17:46:39] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T390531 https://phabricator.wikimedia.org/T394858 https://phabricator.wikimedia.org/T396106) [17:48:35] (03merge) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T390531 https://phabricator.wikimedia.org/T394858 https://phabricator.wikimedia.org/T396106) [17:49:41] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06collaboration-services, 06Traffic: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11846296 (10Umherirrender) Caching is okay as the content of... [17:50:33] (03open) 10dancy: Release 4.250.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1133 [17:53:03] (03merge) 10dancy: Release 4.250.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1133 [17:53:47] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1275985 (owner: 10Fomafix) [17:55:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1275985 (owner: 10Fomafix) [17:57:09] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 13Patch-For-Review: Ensure PipelineLib works on new Jenkins host - https://phabricator.wikimedia.org/T423968#11846351 (10Dzahn) We also need to do something about this in the **p... [17:57:17] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 13Patch-For-Review: Ensure PipelineLib works on new Jenkins host - https://phabricator.wikimedia.org/T423968#11846352 (10Dzahn) a:03Dzahn [18:00:14] !log Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency [18:00:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:01:25] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Automated Supervision test automation for CI - https://phabricator.wikimedia.org/T422555#11846355 (10bd808) An emerging need in the Pretrain project is an understanding of critical user journeys that we can exercise using [[https... [18:04:46] 10Scap, 06serviceops-deprecated: scap leaves mediawiki main releases in the "forward" version after a canary rollback - https://phabricator.wikimedia.org/T375497#11846369 (10dancy) 05Openβ†’03Resolved a:03dancy Fixed and deployed in scap 4.250.0. Now all k8s stages are rolled back to their former stat... [18:07:16] s [18:07:24] ^ i am good at keyboards [18:07:39] I blame the coffee shop. [18:07:48] ...fair [18:10:46] 06Release-Engineering-Team (Seen), 10Scap, 06serviceops-radar, 06SRE: Enable scap to roll back broken changes to MediaWiki - https://phabricator.wikimedia.org/T225207#11846414 (10dancy) 05Openβ†’03Resolved a:03dancy A lot of stuff has happened since this ticket was originally filed. Scap's automat... [18:15:34] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Automated Supervision test automation for CI - https://phabricator.wikimedia.org/T422555#11846435 (10bd808) > renamed this task from [EPIC] Create Critical User Journey automation for CI to [EPIC] Create Automated Supervision tes... [18:17:05] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11846444 (10Aklapper) [18:18:58] 10Scap (SpiderPig πŸ•ΈοΈ): SpiderPig: Give folks the option to rollback changes - https://phabricator.wikimedia.org/T394858#11846455 (10dancy) Some progress on this ticket: As of scap 4.250.0 (deployed), if errors are detected in testservers or canaries stages, all k8s stages are rolled back to their former state.... [18:20:15] 10Scap (SpiderPig πŸ•ΈοΈ): spiderpig should give the revert procedure after a canceled deployment - https://phabricator.wikimedia.org/T396106#11846458 (10dancy) 05Openβ†’03Resolved a:03dancy As of scap 4.250.0 (deployed), if errors are detected in testservers or canaries stages, all k8s stages are rolled bac... [18:20:32] 10Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations, 10VPS-project-Phabricator: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11846463 (10Dzahn) Is this ticket resolved? [18:21:49] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Rebuild all Jenkins agents VM to Bookworm to support Java 21 - https://phabricator.wikimedia.org/T421114#11846483 (10Dzahn) @thcipriani Is this resolved? [18:23:53] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Rebuild all Jenkins agents VM to Bookworm to support Java 21 - https://phabricator.wikimedia.org/T421114#11846495 (10bd808) >>! In T421114#11846483, @Dzahn wrote: > @thcipriani Is this resolved? I'm still working on {T421244}, but will try to wrap i... [18:24:06] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11846497 (10dancy) I've reread this ticket a few times and it's unclear what is needed from scap. Maybe someone can rephrase the request. [18:24:28] (03open) 10aklapper: Enforce MFA for elevated users [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/114 (https://phabricator.wikimedia.org/T304792 https://phabricator.wikimedia.org/T416613) [18:25:44] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: Connect zuul-nodepool to zookeeper and zuul WMCS cluster - https://phabricator.wikimedia.org/T422207#11846507 (10Dzahn) nodepool is connected to zookeeper now afaict: ` [zuul1001:~] $ systemctl status zuul-nodepool .. Apr 17 0... [18:27:45] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Rebuild all Jenkins agents VM to Bookworm to support Java 21 - https://phabricator.wikimedia.org/T421114#11846530 (10Dzahn) @bd808 Oh yea, thank you. I was only thinking of the integration-agent VMs when making that comment. [18:31:15] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11846552 (10Aklapper) [18:35:08] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846567 (10brennen) [18:37:26] 06Release-Engineering-Team, 10Scap, 07Sustainability (Incident Followup): Improve scap/deploy rollback turnaround times - https://phabricator.wikimedia.org/T417166#11846577 (10dancy) >>! In T417166#11808469, @MLechvien-WMF wrote: > Reaching this task as part of triaging incidents from last year. #release-eng... [18:37:32] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review: Remove "Deadline" task subtype - https://phabricator.wikimedia.org/T335395#11846578 (10Aklapper) a:05Aklapperβ†’03None Unlicking the cookie [18:38:19] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846580 (10Dzahn) instance `phabricator-bullseye` already exists. We crea... [18:39:18] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846582 (10Dzahn) @brennen Wanna try to deploy latest to `phabricator-bul... [18:40:39] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846585 (10Dzahn) Ignore the latest comments, mixed up with the even olde... [18:42:22] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846591 (10Dzahn) **phabricator-bookworm:** https://horizon.wikimedia.or... [18:43:36] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846594 (10Dzahn) **phabricator-trixie:** https://horizon.wikimedia.org/... [19:03:34] (03open) 10brennen: Test commit on a private GitLab repo [repos/releng/test-private-repo] - 10https://gitlab.wikimedia.org/repos/releng/test-private-repo/-/merge_requests/1 (https://phabricator.wikimedia.org/T324164) [19:39:54] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 13Patch-For-Review: Ensure PipelineLib works on new Jenkins host - https://phabricator.wikimedia.org/T423968#11846911 (10Dzahn) >>! In T423968#11842096, @thcipriani wrote: > Look... [19:40:16] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06collaboration-services, 13Patch-For-Review: Ensure PipelineLib works on new Jenkins host - https://phabricator.wikimedia.org/T423968#11846915 (10Dzahn) a:05Dzahnβ†’03None [19:43:48] 10Gerrit, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 03Sprint H - For Real: Figure out how to maintain cv repo - given it is primarily committed to by non-WMF - https://phabricator.wikimedia.org/T423961#11846928 (10SBurnett-WMF) [19:48:08] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-devtools, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11846990 (10Dzahn) created a proxy to see what's on the existing VM: http... [20:17:41] !log Running tofu for T421244 [20:17:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:17:43] T421244: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244 [20:18:50] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11847077 (10bd808) `Apply complete! Resources: 6 added, 0 changed, 0 destroyed.` https://gitlab.wikimedia.org/cloudvps-r... [20:27:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-deploy05 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:27:44] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-deploy05 in project deployment-prep - https://phabricator.wikimedia.org/T424091 (10wmcs-alerts) 03NEW [20:30:40] (03open) 10dancy: main.py: Add --backport flag to sync-* implementations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1134 [20:30:44] (03update) 10dancy: main.py: Add --backport flag to sync-* implementations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1134 [20:33:49] (03merge) 10dancy: main.py: Add --backport flag to sync-* implementations [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1134 [20:34:14] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1276037 (https://phabricator.wikimedia.org/T394410) [20:35:31] (03open) 10dancy: Release 4.250.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1135 [20:36:18] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1276037 (https://phabricator.wikimedia.org/T394410) (owner: 10Jforrester) [20:37:28] (03open) 10dancy: Rewrite in Go [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/43 [20:37:31] (03update) 10dancy: Rewrite in Go [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/43 [20:37:46] (03merge) 10dancy: Release 4.250.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1135 [20:37:58] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1276037 (https://phabricator.wikimedia.org/T394410) (owner: 10Jforrester) [20:38:38] !log Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep, for T394410 [20:38:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:38:40] T394410: Move management of recommended Wikifunctions to use CommunityConfiguration - https://phabricator.wikimedia.org/T394410 [20:38:46] (03update) 10dancy: Rewrite in Go [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/43 [20:38:50] (03update) 10dancy: Rewrite in Go [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/43 [20:41:49] (03merge) 10dancy: Rewrite in Go [repos/releng/k8s-pvc-cleaner] - 10https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_requests/43 [20:44:00] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11847209 (10bd808) * wait for first puppet run so that ssh works as a project member * ssh deployment-deploy05.deployment-prep.eqiad1.wikimedi... [20:49:46] (03open) 10dancy: k8s-pvc-cleaner/k8s-pvc-cleaner-values.yaml.tftpl: Bump to v2.0.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/578 [20:49:49] (03update) 10dancy: k8s-pvc-cleaner/k8s-pvc-cleaner-values.yaml.tftpl: Bump to v2.0.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/578 [20:50:58] (03merge) 10dancy: k8s-pvc-cleaner/k8s-pvc-cleaner-values.yaml.tftpl: Bump to v2.0.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/578 [20:52:43] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11847282 (10bd808) `lang=shell-session,COUNTEREXAMPLE bd808@deployment-deploy05:~$ sudo -i puppet agent -tv Info: Using environment 'productio... [21:30:42] (03open) 10dancy: k8s-pvc-cleaner/k8s-pvc-cleaner-values.yaml.tftpl: Disable dryRun mode [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/579 [21:30:46] (03update) 10dancy: k8s-pvc-cleaner/k8s-pvc-cleaner-values.yaml.tftpl: Disable dryRun mode [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/579 [21:31:55] (03merge) 10dancy: k8s-pvc-cleaner/k8s-pvc-cleaner-values.yaml.tftpl: Disable dryRun mode [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/579 [21:34:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:34:39] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100 (10wmcs-alerts) 03NEW [21:37:00] ^^ looking at this now, sorry for the spam [21:43:47] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100#11847477 (10bking) 05Openβ†’03In progress p:05Triageβ†’03Medium a:03bking [21:50:55] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 06Discovery-Search (2026.03.03 - 2026.04.03): Migrate beta cluster to OpenSearch 2.x - https://phabricator.wikimedia.org/T421763#11847518 (10A_smart_kitten) [21:54:32] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch13 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:56:59] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-03-27 - 2026-04-17): No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100#11847571 (10bd808) [21:57:00] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 06Discovery-Search (2026.03.03 - 2026.04.03): Migrate beta cluster to OpenSearch 2.x - https://phabricator.wikimedia.org/T421763#11847572 (10bd808) [21:57:29] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-deploy05 in project deployment-prep - https://phabricator.wikimedia.org/T424091#11847573 (10bd808) 05Openβ†’03In progress a:03bd808 [21:59:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:59:38] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep - https://phabricator.wikimedia.org/T424104 (10wmcs-alerts) 03NEW [22:08:16] 06Release-Engineering-Team (Priority Backlog πŸ“₯): Talk with Security/Product Safety and Integrity foks about Pretrain workflow/timing implications - https://phabricator.wikimedia.org/T424107 (10bd808) 03NEW [22:15:43] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-03-27 - 2026-04-17), 06Discovery-Search (2026.03.03 - 2026.04.03): Migrate beta cluster to OpenSearch 2.x - https://phabricator.wikimedia.org/T421763#11847655 (10bking) Update before the Earth Day holiday: I changed the hieradata, but Puppet didn't li... [22:28:41] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11847695 (10bd808) `lang=shell-session,counterexample bd808@deployment-deploy05:~$ sudo -i puppet agent -tv Info: Using environment 'productio... [22:29:29] (03open) 10brennen: WIP: hide events from non-public projects [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 [22:33:10] Project beta-code-update-eqiad build #597127: 04FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/597127/ [22:45:22] Yippee, build fixed! [22:45:22] Project beta-code-update-eqiad build #597128: 09FIXED in 2 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/597128/ [23:02:32] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-deploy05 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:07:20] (03update) 10brennen: WIP: hide events from non-public projects [repos/releng/gitlab-webhooks] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 [23:14:14] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1276074 (https://phabricator.wikimedia.org/T422031) [23:14:25] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1276074 (https://phabricator.wikimedia.org/T422031) (owner: 10Jforrester) [23:16:06] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1276074 (https://phabricator.wikimedia.org/T422031) (owner: 10Jforrester) [23:17:32] !log Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs, for T422031 [23:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:17:35] T422031: Implement large/small buttons in OOUI - https://phabricator.wikimedia.org/T422031 [23:24:01] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too [integration/config] - 10https://gerrit.wikimedia.org/r/1276077 (https://phabricator.wikimedia.org/T394410) [23:24:11] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too [integration/config] - 10https://gerrit.wikimedia.org/r/1276077 (https://phabricator.wikimedia.org/T394410) (owner: 10Jforrester) [23:25:50] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too [integration/config] - 10https://gerrit.wikimedia.org/r/1276077 (https://phabricator.wikimedia.org/T394410) (owner: 10Jforrester) [23:26:19] !log Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too, for T394410 [23:26:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:26:22] T394410: Move management of recommended Wikifunctions to use CommunityConfiguration - https://phabricator.wikimedia.org/T394410 [23:42:59] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11847986 (10bd808) Now that I have beaten the initial puppet run into submission, it is time to do the puppetserver swit... [23:50:33] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11847996 (10bd808) `lang=irc,name=#wikimedia-sre-collab ο»Ώο»Ώ[23:47] < bd808> mutante: we have a problem with the Jenki... [23:52:18] 10Continuous-Integration-Config, 10DiscussionTools, 10MobileFrontend: DiscussionTools gate-and-submit jobs don't run tests that depend on MobileFrontend - https://phabricator.wikimedia.org/T424113 (10matmarex) 03NEW [23:54:49] 10Continuous-Integration-Config, 10DiscussionTools, 10MobileFrontend: DiscussionTools gate-and-submit jobs don't run tests that depend on MobileFrontend - https://phabricator.wikimedia.org/T424113#11848011 (10Jdforrester-WMF) I think this was broken by 4f57772b019a6ba18e7a27e7ff7f1ae97ebeaff6 where Antoine w... [23:55:43] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/DiscussionTools] Add MF dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1276083 (https://phabricator.wikimedia.org/T424113) [23:58:59] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/DiscussionTools] Add MF dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1276083 (https://phabricator.wikimedia.org/T424113) (owner: 10Jforrester)