[00:24:12] this commit failed 6 gate checks despite the console of those checks apparently showing success: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1293827 [00:24:46] anyone know what is going on with that? [00:28:52] Expand the "PostBuildScript" [00:29:17] * Reedy waits for phab [00:29:37] hmm, right, still kind of vague though [00:30:18] Not sure on the specific task, but castor is a source of a few different failures [00:30:31] T419488 [00:30:32] T419488: PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488 [00:30:43] the timestamps tell a story [00:30:44] I guess that exactly describes the behaviour you're seeing [00:30:44] 00:03:37.518 Waiting for the completion of castor-save-workspace-cache [00:30:44] 00:08:18.355 Build step 'Execute scripts' changed build result to FAILURE [00:31:03] almost a 5 minute delay [00:32:52] ok after reading that I guess I am just meant to run it again [00:33:12] unfortunately [00:33:41] with the ci cache things... if things haven't changed, patches not merged to the repo in the meantime, rechecking should be faster than a full run [04:11:45] (03Abandoned) 10Baran: Add Pashto (ps) translations for Gadgets namespaces and special pages [extensions/Gadgets] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1295795 (owner: 10Baran) [08:07:11] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (Basel 26): Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#11975289 (10hashar) [08:13:51] (03open) 10kbach: Re-enable OAD example warnings test [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/9 [08:14:15] (03PS6) 10Arnaudb: zuul: retry policy on network errors [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) [08:16:01] (03CR) 10Arnaudb: "thanks for the +1 @adancy@wikimedia.org and the rebase @tcipriani@wikimedia.org" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [08:39:46] 10Gerrit, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review, 07Puppet: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#11975411 (10ABran-WMF) >>! In T420184#11968357, @Dzahn wrote: > The stri... [11:30:18] (03open) 10kbach: Clarify license for OAD example [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/10 [11:59:07] (03PS1) 10Ozge: feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [11:59:49] (03PS2) 10Ozge: feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [12:01:28] (03CR) 10CI reject: [V:04-1] feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [12:09:42] (03PS3) 10Ozge: feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [12:32:07] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488#11976154 (10matmarex) This is still happening, e... [13:15:18] Can someone give zuul a kick? [13:15:34] https://integration.wikimedia.org/zuul/ is showing jobs in progress that are already completed in jenkins [13:16:16] Specifically https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-selenium/57912/console is appearing as still in progress in zuul [13:17:36] hashar: ^ do you think zuul will recover on its own? or should we take a look? [13:21:46] Last execution of the castor job was at 14:02: https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ [13:21:56] *13:02 UTC [13:22:17] I would suggest it's not going to recover by itself? [13:24:13] 20m ago, yeah, it doesn't look good. hashar should be around, let's see if he responds. I've never restarted zuul myself and it worries me to do that without him [13:28:53] Hmm, doesn't seem to be around [13:28:57] I can ping in Slack? [13:29:23] Or James_F ^^ [13:32:48] pinged in Slack [13:32:54] Thanks [13:33:11] It says they are away in slack tho... [13:34:21] ok, I'm going to try a restart [13:34:34] Thanks, good luck :D [13:38:54] that seemed to have flushed the queue, zuul is picking up new jobs [13:39:00] I still see the castor jobs on CI blocked, I don't know what that's about [13:39:12] I'll keep trying to poke, hopefully not break anything [13:53:41] I can't cancel the castor jobs with the UI and I can't force cancel them with the admin console, awesome [13:54:28] dear releng, I am getting a NOT_REGISTERED in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1294271 [13:54:37] 3 times in a row [13:55:17] (ditto, but appears folks are aware) [13:56:27] alright then [14:01:20] (03CR) 10AikoChou: feat: adds editing suggestions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:03:35] (03PS4) 10Ozge: feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [14:03:49] (03CR) 10CI reject: [V:04-1] feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:04:04] (03CR) 10Ozge: "fixing review comments." [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:04:25] (03CR) 10Ozge: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:04:34] (03CR) 10Ozge: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:05:50] !log trying to reconnect Gearman [14:05:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:07:35] (03CR) 10Ozge: "test" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:07:41] (03CR) 10Ozge: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:08:11] (03CR) 10AikoChou: "Also, the commit message could be a bit clearer. Maybe we could follow this format:" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:08:23] (03CR) 10Reedy: "repeatedly doing "recheck" is probably not going to solve the problem that is unrelated to your patch" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:08:28] (03CR) 10Thiemo Kreuz (WMDE): [C:04-1] "What is this supposed to do?" [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1295911 (owner: 10Baran) [14:08:54] (03Abandoned) 10Thiemo Kreuz (WMDE): Review access change [extensions/Gadgets] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1295890 (owner: 10Baran) [14:09:02] (03Abandoned) 10Thiemo Kreuz (WMDE): Review access change [extensions/Gadgets] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1295844 (owner: 10Baran) [14:12:01] !log option to "Enable Gearman" times out. Can't re-enable from UI. Gearman plugin logs are empty. Neat [14:12:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:13:01] I see the CI issue is something bigger. I'll consider the backport window as something that won't happen and reschedule the patches that were scheduled for today's UTC afternoon [14:17:21] (03PS5) 10Ozge: feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [14:17:29] (03CR) 10CI reject: [V:04-1] feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:23:11] !log trying to reconnect castor node, see if that helps somehow [14:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:24:28] (03PS2) 10Hashar: dockerfiles: add support for Pypy Python 3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1295865 (https://phabricator.wikimedia.org/T423607) [14:24:31] (03CR) 10Ozge: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:24:36] (03CR) 10CI reject: [V:04-1] dockerfiles: add support for Pypy Python 3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1295865 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [14:24:55] (03CR) 10Hashar: [C:03+2] "Amended to rebuild all the images depending on `python-all`" [integration/config] - 10https://gerrit.wikimedia.org/r/1295865 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [14:25:05] (03CR) 10CI reject: [V:04-1] dockerfiles: add support for Pypy Python 3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1295865 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [14:25:25] (03CR) 10Ozge: "not sure what happened but I think the jenkins tests has stopped getting triggered." [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [14:26:53] (03PS1) 10Hashar: jjb: update python jobs for Pypy/Python3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1296604 (https://phabricator.wikimedia.org/T423607) [14:27:03] (03CR) 10CI reject: [V:04-1] jjb: update python jobs for Pypy/Python3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1296604 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [14:28:28] !log bring back castor node, that didn't help [14:28:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:32:16] jnuche: what is broken? :\ [14:33:17] hashar: everything? :) I was about to try restarting Jenkins [14:33:49] ah [14:34:10] (03open) 10aklapper: Put back our custom robots.txt additions [repos/phabricator/phabricator] (work/2026-06-01-merge-phorge) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/118 [14:34:36] (03merge) 10aklapper: Put back our custom robots.txt additions [repos/phabricator/phabricator] (work/2026-06-01-merge-phorge) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/118 [14:34:48] apparently there are bunch of builds stalling on https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ [14:35:05] hashar: call? https://meet.google.com/gcc-cyqe-jzs [14:35:09] (03update) 10aklapper: Merge upstream master as of 2026-06-01 [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/117 (https://phabricator.wikimedia.org/T410849) (owner: 10brennen) [14:35:24] with https://integration.wikimedia.org/ci/computer/integration-castor06/ having a stall [14:35:24] yeah [14:35:27] let me switch connection [14:38:41] !log restarting Jenkins [14:38:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:45:05] (03open) 10aklapper: Draft: Reduce our downstream diff in PhabricatorElasticFulltextStorageEngine [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/119 [14:50:49] (03CR) 10Hashar: [C:03+2] "CI had some issue ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/1295865 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [14:52:30] (03Merged) 10jenkins-bot: dockerfiles: add support for Pypy Python 3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1295865 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [14:53:18] (03open) 10aklapper: Draft: Conduit: Improve validation of custom UserQuery input [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/65 [14:56:49] folks, if you're still seeing NOT_REGISTERED on your patches, you can try to rerun/recheck and that should now work [14:57:08] Thank, mine works now [14:57:26] !log Jenkins/Zuul is back [14:57:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:57:38] Thanks! [14:58:54] I see our old friend β€œGnuTLS recv error (-54): Error in the pull function” is still around though πŸ˜” https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-selenium/57916/console [14:59:01] but yay for CI being less broken [15:02:36] so the jobs in Jenkins were stall waiting for the castor cache to be saved [15:02:50] and there is only one build of it that can happen [15:03:13] due to some in Jenkins, a build of it could not start on the node cause Jenkins considered it to not be able to run it [15:03:32] thus all jobs got stalled waiting for it to start [15:04:23] good news is phendeskog flagged the castor-save-workspace-cache being a bottleneck/slow etc sso we have discussions to have it trigger less often [15:04:28] (and make it faster) [15:05:31] 10Gerrit, 06collaboration-services: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11976938 (10hashar) 05Openβ†’03Stalled Marking stalled pending T418215 which itself is pending an upgrade of Gerrit to 3.11.8 or later (IIRC). [15:06:21] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (Tallinn 27): Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#11976949 (10AMarkossyan-WMF) [15:06:50] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Tallinn 27): POC: Try to make Castor save async (to remove queue time from job times) - https://phabricator.wikimedia.org/T425411#11976958 (10AMarkossyan-WMF) [15:07:37] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Tallinn 27): Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822#11976968 (10AMarkossyan-WMF) [15:08:43] (03open) 10aklapper: Draft: Remove some downstream diff in PhabricatorElasticFulltextStorageEngine [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/120 [15:09:33] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Critical User Journey test automation for CI - https://phabricator.wikimedia.org/T422555#11977004 (10SLong-WMF) [15:13:57] (03open) 10jnuche: staging/staging.tfvars: `buildkitd_tag` to `wmf-v0.30.0` [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/611 [15:15:18] 10Continuous-Integration-Infrastructure, 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258#11977056 (10hashar) Finally I have managed to test it! I cherry picked yo... [15:15:55] 10Phabricator, 10Wikimedia-Phabricator-Extensions: Enhance custom Conduit endpoint policy.create for Named Policies support - https://phabricator.wikimedia.org/T427934 (10Aklapper) 03NEW p:05Triageβ†’03Low [15:15:59] (03merge) 10jnuche: staging/staging.tfvars: `buildkitd_tag` to `wmf-v0.30.0` [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/611 [15:16:06] 10Phabricator, 10Wikimedia-Phabricator-Extensions: Enhance custom Conduit endpoint policy.create for Named Policies support - https://phabricator.wikimedia.org/T427934#11977089 (10Aklapper) [15:16:14] 10Continuous-Integration-Infrastructure, 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258#11977087 (10hashar) I also cherry-picked https://gerrit.wikimedia.org/r/c/... [15:16:20] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review: Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11977090 (10Aklapper) [15:16:24] 06Release-Engineering-Team (Radar), 10QS-Test-Automation, 07Epic: [EPIC] Create Critical User Journey test automation for CI - https://phabricator.wikimedia.org/T422555#11977092 (10SLong-WMF) [15:16:33] (03open) 10dancy: README.md: Clarify deployment instructions [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/80 [15:16:45] (03update) 10dancy: README.md: Clarify deployment instructions [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/80 [15:18:57] (03update) 10dancy: README.md: Clarify deployment instructions [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/80 [15:24:48] !log Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/1295865 # T423607 [15:24:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:24:50] T423607: Add PyPy 3.11 to Wikimedia CI - https://phabricator.wikimedia.org/T423607 [15:26:08] 10Continuous-Integration-Infrastructure, 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Puppet fail to create volume group for ephemeral disk space when it is sda (instead of sdb) - https://phabricator.wikimedia.org/T422258#11977210 (10Andrew) 05Openβ†’03Resolved [15:27:02] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T423914#11977236 (10Jakob_WMDE) [15:33:01] (03approved) 10jnuche: README.md: Clarify deployment instructions [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/80 (owner: 10dancy) [15:33:17] (03merge) 10dancy: README.md: Clarify deployment instructions [repos/releng/buildkit] (wmf/build) - 10https://gitlab.wikimedia.org/repos/releng/buildkit/-/merge_requests/80 [15:38:40] (03open) 10jnuche: production/prod.tfvars: `buildkitd_tag` to `wmf-v0.30.0` [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/612 [15:40:30] (03merge) 10jnuche: production/prod.tfvars: `buildkitd_tag` to `wmf-v0.30.0` [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/612 [15:41:40] 10Continuous-Integration-Config, 06Release-Engineering-Team (Seen), 10Librarization, 10Quibble, 13Patch-For-Review: Install require-dev composer dependencies from extension repos in "vendor" quibble-phpunit job - https://phabricator.wikimedia.org/T220723#11977432 (10Krinkle) [15:46:09] 10Continuous-Integration-Config, 06Release-Engineering-Team (Seen), 10Librarization, 10Quibble, 13Patch-For-Review: Install require-dev composer dependencies from extension repos in "vendor" quibble-phpunit job - https://phabricator.wikimedia.org/T220723#11977486 (10Jdforrester-WMF) @Krinkle: The new tit... [15:49:04] !log Updating buildkitd to v0.30.0 in gitlab-cloud-runners (T426212) [15:49:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:49:07] T426212: Buildkit v0.30.0 released - https://phabricator.wikimedia.org/T426212 [15:49:34] 10Phabricator, 10Wikimedia-Phabricator-Extensions: Enhance custom Conduit endpoint policy.create for Named Policies support - https://phabricator.wikimedia.org/T427934#11977513 (10Aklapper) Completely untested and many TODOs, not even WIP: ` diff --git a/src/conduit/CreatePolicyConduitAPIMethod.php b/src/cond... [15:54:20] (03CR) 10Hashar: [C:03+2] "Jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/1296604 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [15:57:20] (03Merged) 10jenkins-bot: jjb: update python jobs for Pypy/Python3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1296604 (https://phabricator.wikimedia.org/T423607) (owner: 10Hashar) [16:04:45] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Add PyPy 3.11 to Wikimedia CI - https://phabricator.wikimedia.org/T423607#11977651 (10hashar) I have build the new images which have Pypy 7.3.20 which would match Python 3.11.13. All the Jenkins jobs have been... [16:05:42] !log integration: upgraded pypy from 7.3.11 (py3.9) to 7.3.20 (py3.11) # T423607 [16:05:43] 10Gerrit, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review, 07Puppet: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#11977655 (10Dzahn) Ah, it's in the puppetserver module. Thanks! The pat... [16:05:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:05:45] T423607: Add PyPy 3.11 to Wikimedia CI - https://phabricator.wikimedia.org/T423607 [16:06:03] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Add PyPy 3.11 to Wikimedia CI - https://phabricator.wikimedia.org/T423607#11977657 (10hashar) 05Openβ†’03Resolved [16:10:51] 10Continuous-Integration-Config, 06Release-Engineering-Team (Seen), 10Librarization, 10Quibble, 13Patch-For-Review: Install require-dev composer dependencies from extension repos in "vendor" quibble-phpunit job - https://phabricator.wikimedia.org/T220723#11977696 (10Krinkle) I agree with you that install... [16:21:49] (03open) 10dancy: Equalize Reggie's memory request and limit [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/613 [16:21:51] (03update) 10dancy: Equalize Reggie's memory request and limit [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/613 [16:37:12] (03update) 10dancy: Equalize Reggie's memory request and limit [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/613 [16:37:16] (03update) 10dancy: Equalize Reggie's memory request and limit [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/613 [16:38:26] (03merge) 10dancy: Equalize Reggie's memory request and limit [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/613 [16:38:50] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.30.0 released - https://phabricator.wikimedia.org/T426212#11978001 (10dancy) 05Openβ†’03Resolved a:03dancy Buildkit v0.30.0 deployed to all places. [16:49:50] (03CR) 10Dzahn: "pinged on slack to see if we can get this merged" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [17:01:03] (03CR) 10Ozge: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [17:06:26] (03PS6) 10Ozge: feat: adds editing suggestions [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [17:15:23] (03PS7) 10Ozge: inference-services: Add LLM generated editing suggestions CI/CD pipelines. [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [17:15:38] (03PS8) 10Ozge: inference-services: Add LLM generated editing suggestions CI/CD pipelines. [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [17:39:24] 06Release-Engineering-Team, 10Catalyst (PatchDemo): patchdemo: Move "delete" action into a Slim action, template, etc. - https://phabricator.wikimedia.org/T427964 (10brennen) 03NEW [17:46:32] (03CR) 10AikoChou: [C:03+1] "Two nits, otherwise looks good to me!" [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [17:48:41] 06Release-Engineering-Team (Priority Backlog πŸ“₯): [FY25-26 WE6.7.3] Automated supervision - https://phabricator.wikimedia.org/T427968 (10bd808) 03NEW [17:50:07] 06Release-Engineering-Team (Priority Backlog πŸ“₯): [FY25-26 WE6.7.3] Automated supervision - https://phabricator.wikimedia.org/T427968#11978412 (10bd808) p:05Triageβ†’03Medium [18:01:20] 10Phabricator, 06Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review: Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11978478 (10brennen) 05Openβ†’03In progress [18:01:48] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review: Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11978499 (10brennen) [18:13:22] (03open) 10brennen: update submodules for 2026-06-09 deploy / upstream merge [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/106 (https://phabricator.wikimedia.org/T410849) [18:13:29] !log devtools phab/phorge: deployed work/2026-06-01-merge-phorge to https://phabricator.wmcloud.org/ for testing (T410849) [18:13:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:13:33] T410849: Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849 [18:25:54] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T423914#11978639 (10dancy) Train is blocked at testwikis due to T427935. [18:29:45] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10QS-Test-Automation, 13Patch-For-Review, 06Quality-and-Test-Engineering-Team (Quality Engineering): Better prevent changes in MW core from breaking WMF production extensions - https://phabricator.wikimedia.org/T404403#11978662 (10aude)... [18:32:03] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10QS-Test-Automation, 13Patch-For-Review, 06Quality-and-Test-Engineering-Team (Quality Engineering): Better prevent changes in MW core from breaking WMF production extensions - https://phabricator.wikimedia.org/T404403#11978672 (10aude)... [18:52:58] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T423914#11978730 (10dancy) >>! In T423914#11978634, @dancy wrote: > Train is blocked at testwikis due to T427935. Re... [19:21:16] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T423914#11978797 (10Jdforrester-WMF) [19:27:55] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Use encrypted rsync for Gerrit - https://phabricator.wikimedia.org/T412780#11978808 (10Dzahn) 05Openβ†’03Resolved I think this is it :) [19:39:36] (03PS1) 10Hashar: Zuul: drop browser tests from GrowthExperiments wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/1296667 (https://phabricator.wikimedia.org/T417412) [19:40:21] (03CR) 10Hashar: [C:03+2] Zuul: use standalone jobs for GrowthExperiments Cypress tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1268559 (https://phabricator.wikimedia.org/T417412) (owner: 10Zfilipin) [19:42:08] 10Beta-Cluster-Infrastructure, 06Growth-Team, 10GrowthExperiments-NewcomerTasks: [beta-cluster] Fetching task suggestions failed: cirrussearch-backend-error - https://phabricator.wikimedia.org/T427196#11978855 (10Michael) This does not seem to be primarily a GrowthExperiments problem, though we are affected. [19:42:53] (03CR) 10Michael Große: [C:03+1] Zuul: drop browser tests from GrowthExperiments wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/1296667 (https://phabricator.wikimedia.org/T417412) (owner: 10Hashar) [20:19:12] (03CR) 10Thcipriani: [C:03+2] "Noting that after merge, needs a subsequent [quibble release](https://doc.wikimedia.org/quibble/releasing.html) and rebuild of images for " [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [20:19:48] (03CR) 10Jforrester: [C:03+1] Zuul: drop browser tests from GrowthExperiments wmf branches [integration/config] - 10https://gerrit.wikimedia.org/r/1296667 (https://phabricator.wikimedia.org/T417412) (owner: 10Hashar) [20:36:25] 10Beta-Cluster-Infrastructure, 06Discovery-Search, 06Growth-Team, 10GrowthExperiments-NewcomerTasks: [beta-cluster] Fetching task suggestions failed: cirrussearch-backend-error - https://phabricator.wikimedia.org/T427196#11979062 (10Urbanecm) [20:37:13] (03Merged) 10jenkins-bot: zuul: retry policy on network errors [integration/quibble] - 10https://gerrit.wikimedia.org/r/1278483 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [20:50:44] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865#11979145 (10Dzahn) The patch by Arnaud to add retries ha... [20:54:30] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10QS-Test-Automation, 13Patch-For-Review, 06Quality-and-Test-Engineering-Team (Quality Engineering): Better prevent changes in MW core from breaking WMF production extensions - https://phabricator.wikimedia.org/T404403#11979182 (10SLong-... [21:01:56] 10Gerrit: Gerrit marks patches as having merge conflicts after their parent is merged - https://phabricator.wikimedia.org/T427696#11979235 (10SomeRandomDeveloper) That change was deployed 11 months ago though, and I've only noticed the bug very recently (in the last two weeks). It's also possible that I missed i... [21:12:35] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Tallinn 27): Investigate multiple npm cache folders for mwext-codehealth-master-non-voting - https://phabricator.wikimedia.org/T427822#11979266 (10Peter) Hi @Jdforrester-WMF do you remember the reason we set the cache for npm differently in... [21:22:09] (03PS9) 10Ozge: inference-services: Add LLM generated editing suggestions CI/CD pipelines. [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) [21:22:49] (03CR) 10Ozge: inference-services: Add LLM generated editing suggestions CI/CD pipelines. (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [21:24:44] (03CR) 10Ozge: "thank you @achou@wikimedia.org. addressed the comments. would you mind +1 or +2." [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [21:25:41] (03CR) 10Ozge: "would you mind +1 or +2 again." [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [21:26:01] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11979326 (10BCornwall) a:05BCornwallβ†’03Vgutierrez [21:26:07] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11979328 (10BCornwall) 05In progressβ†’03Stalled [21:26:18] (03CR) 10Ozge: [C:03+1] "I don't have +2 permissions and I'm not sure who has it." [integration/config] - 10https://gerrit.wikimedia.org/r/1296559 (https://phabricator.wikimedia.org/T427794) (owner: 10Ozge) [21:43:39] (03update) 10kineticpelagic: Re-enable OAD example warnings test [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/9 (owner: 10kbach) [21:44:04] (03approved) 10kineticpelagic: Re-enable OAD example warnings test [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/9 (owner: 10kbach) [21:44:29] (03merge) 10kineticpelagic: Re-enable OAD example warnings test [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/9 (owner: 10kbach) [21:48:58] (03PS1) 10Thcipriani: Changelog: 1.18.1 release [integration/quibble] - 10https://gerrit.wikimedia.org/r/1296685 [21:49:28] (03update) 10kineticpelagic: Clarify license for OAD example [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/10 (owner: 10kbach) [21:50:36] (03open) 10ejegg: Create trixie image for fundraising ML services [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/107 [21:50:37] (03Abandoned) 10Phedenskog: castor: Only save cache on postmerge [integration/config] - 10https://gerrit.wikimedia.org/r/1289280 (https://phabricator.wikimedia.org/T426729) (owner: 10Phedenskog) [22:45:21] (03CR) 10Thcipriani: [C:03+2] Changelog: 1.18.1 release [integration/quibble] - 10https://gerrit.wikimedia.org/r/1296685 (owner: 10Thcipriani) [23:02:22] (03Merged) 10jenkins-bot: Changelog: 1.18.1 release [integration/quibble] - 10https://gerrit.wikimedia.org/r/1296685 (owner: 10Thcipriani) [23:10:56] !log tag quibble 1.18.1 @ 4b7959553c095d811426f394165c69ecc13a44eb [23:10:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:12:53] (03PS1) 10Thcipriani: release: Start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1296694 [23:16:17] (03PS1) 10Thcipriani: dockerfiles: update Quibble to 1.18.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1296695 [23:19:36] (03CR) 10Thcipriani: [C:03+2] dockerfiles: update Quibble to 1.18.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1296695 (owner: 10Thcipriani) [23:21:21] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.18.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1296695 (owner: 10Thcipriani) [23:23:59] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1296695 [23:23:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL