[00:07:14] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, and 3 others: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#11853530 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitile... [00:22:32] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-deploy05 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [00:52:32] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-deploy05 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [00:53:32] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [00:53:40] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T424288 (10wmcs-alerts) 03NEW [00:54:36] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11853603 (10bd808) 05Open→03Declined Declined in favor of {T256168} per our finding of PHP issues for a quick sw... [01:07:47] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, and 3 others: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#11853630 (10bd808) The new `/usr/local/bin/wmf-beta-update-all`... [01:09:09] https://phabricator.wikimedia.org/T256168#11853630 is the current state of MediaWiki code update jobs in Beta Cluster. TL;DR is that Jenkins jobs are disabled and a systemd timer on deployment-deploy04 is in charge. [01:10:49] thcipriani, Reedy, James_F: ^ hopefully just FYI. I'm off until Monday, but if things get weird any of y'all can yell for my help. (Same for anyone who reads this in backlog) [01:11:10] Ack. [02:18:32] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-eventgate-4 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [03:33:36] 10GitLab: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295 (10Count_Count) 03NEW [03:43:56] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11853858 (10RLazarus) Suppose version A is running, and we're deploying version B. Helm enforces a timeout: after starting up, if version B still isn't re... [04:45:55] 10Beta-Cluster-Infrastructure, 07Chinese-Sites: 2FA removal and request for sysop and interfaceadmin on zh.wikipedia.beta.wmcloud.org - https://phabricator.wikimedia.org/T423830#11853965 (10amanojaku) Hello, I've contacted the T&S team to resolve my account issues. I'll provide an update once there is further... [07:23:01] 10Beta-Cluster-Infrastructure, 07Chinese-Sites: 2FA removal and request for sysop and interfaceadmin on zh.wikipedia.beta.wmcloud.org - https://phabricator.wikimedia.org/T423830#11854169 (10jeremyb) 05Open→03Stalled [08:16:04] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Title of commit message cut off in Diffusion after a full stop - https://phabricator.wikimedia.org/T130557#11854313 (10Aklapper) 05Open→03Stalled [08:17:21] (03update) 10aklapper: Notify Phab WMF users to link their SUL account [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/104 (https://phabricator.wikimedia.org/T407064) [08:32:30] Project mediawiki-core-doxygen build #19886: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/19886/ [08:36:19] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11854362 (10ABran-WMF) Envoy's log does not show anymore [[ https://envoy-insider.mygraphql.com/en/latest/arch/buffer/bu... [08:38:20] 10Phabricator, 06Release-Engineering-Team: Phabricator "estimated points remaining" custom field - https://phabricator.wikimedia.org/T421974#11854366 (10Aklapper) 05Stalled→03Declined Unfortunately closing this Phabricator task as no further information has been provided. After more information has be... [09:05:18] 10Gerrit, 06collaboration-services, 07Documentation, 13Patch-For-Review, 07Sustainability (Incident Followup): Update and improve operation runbooks and documentation for Gerrit - https://phabricator.wikimedia.org/T423601#11854423 (10ABran-WMF) [09:17:42] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 06Discovery-Search (2026.03.03 - 2026.04.03): Migrate beta cluster to OpenSearch 2.x - https://phabricator.wikimedia.org/T421763#11854500 (10Gehel) [09:19:29] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-04-24 - 2026-05-15): No Puppet resources found on instance deployment-cirrussearch14 on project deployment-prep - https://phabricator.wikimedia.org/T424100#11854542 (10Gehel) [09:23:24] 10Gerrit, 06collaboration-services, 07Documentation, 13Patch-For-Review, 07Sustainability (Incident Followup): Update and improve operation runbooks and documentation for Gerrit - https://phabricator.wikimedia.org/T423601#11854571 (10ABran-WMF) [09:32:47] Yippee, build fixed! [09:32:47] Project mediawiki-core-doxygen build #19887: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/19887/ [09:39:29] 10GitLab: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11854609 (10Count_Count) Hmm, `docker pull docker-hub-mirror.cloud.releng.team/library/rust:latest` works for me locally. Not sure what is going on. [10:14:31] morning, in the absence of hasharAway I am curious if there's someone around who I could try to convince to apply https://gerrit.wikimedia.org/r/c/integration/config/+/1276428 ideally during European day time? [10:14:53] in essence my intent is to have CI run a "npm test" on a new Gerrit repo [10:15:22] test script involves some cypress test running hence I went for the -browser template [10:16:01] I anticipate it might not fully work but I'd like to see what it might complain about (previously the repo was github hosted hence some CI environment different might occur) [11:51:42] leszek_wmde: Yeah, I can do that in a bit this afternoon [12:23:47] cheers Reedy [12:24:00] leszek_wmde: Can do it now if you want? I'm sat at my desk [12:25:31] I guess it's against a specific repo, so even if it doesn't work... [12:25:32] * Reedy deploys [12:25:39] (03CR) 10Reedy: [C:03+2] Set up CI for wikibase/new-lexeme-special-page repository [integration/config] - 10https://gerrit.wikimedia.org/r/1276428 (https://phabricator.wikimedia.org/T424098) (owner: 10WMDE-leszek) [12:27:26] (03Merged) 10jenkins-bot: Set up CI for wikibase/new-lexeme-special-page repository [integration/config] - 10https://gerrit.wikimedia.org/r/1276428 (https://phabricator.wikimedia.org/T424098) (owner: 10WMDE-leszek) [12:27:54] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276428 [12:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:29:03] https://gerrit.wikimedia.org/r/c/wikibase/new-lexeme-special-page/+/1277084 [12:29:48] https://integration.wikimedia.org/ci/job/generic-node20-browser/5489/console [12:30:44] exciting [12:31:53] >13:31:23 Finished: SUCCESS [12:31:56] amazing [12:32:06] wow, I didn't anticipate that [12:32:16] thanks very much Reedy, job done [12:33:45] I did notice some very important warnings [12:33:52] 13:30:35 /src/src/components/SpellingVariantInput.vue [12:33:52] 13:30:35 136:1 warning This line has a length of 105. Maximum allowed is 100 max-len [12:34:08] Want me to +2 https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/1276811 ? [12:34:28] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Plovdiv 25): Investigate if there's a way to make castor wait time smaller - https://phabricator.wikimedia.org/T423557#11855203 (10Peter) Let me try to get more data from what we actually have in Jenkins. Then the nex... [12:41:14] Reedy: I've just asked WMDE developers but if you feel like it - sure [12:52:30] do you want LibUp on that repo? [12:53:06] I'm guessing it maybe have a github equivalent before... leszek_wmde ^^ [12:54:08] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Lexicographical data, 10wmde-wikidata-tech, 13Patch-For-Review: Move Wikibase Lexeme's New Lexeme Special Page component's repository to Wikimedia Gerrit - https://phabricator.wikimedia.org/T424098#11855264 (10WMDE-leszek) [13:18:39] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 13Patch-For-Review: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244#11855344 (10Dzahn) How about using the sury.org packages as is recommended in T397075#11655976 ? We could add the s... [13:57:08] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Plovdiv 25): Investigate if there's a way to make castor wait time smaller - https://phabricator.wikimedia.org/T423557#11855555 (10Peter) Ok I parsed some Jenkins logs. These are wait times from the last runs: | job... [14:59:34] 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11855926 (10dancy) [15:22:02] 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11855990 (10dancy) I ran `dmesg` on the node that is current running the docker-hub-mirror pod and I see that the nginx process has been killed several times due to OOM... [15:25:48] (03open) 10dancy: Bump docker-hub-mirror nginx memory requests/limits to 100Mi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/586 (https://phabricator.wikimedia.org/T424295) [15:25:50] (03update) 10dancy: Bump docker-hub-mirror nginx memory requests/limits to 100Mi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/586 (https://phabricator.wikimedia.org/T424295) [15:27:02] (03merge) 10dancy: Bump docker-hub-mirror nginx memory requests/limits to 100Mi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/586 (https://phabricator.wikimedia.org/T424295) [15:37:10] 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856063 (10dancy) 05Open→03Resolved a:03dancy I doubled the memory request/limit for the nginx container. @Count_Count Thanks for the report. This should... [15:40:24] 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856086 (10Count_Count) @dancy Works now, thanks! [15:41:21] 10GitLab, 06Release-Engineering-Team: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856091 (10dancy) 05Resolved→03Open I just saw another OOM kill so I'm making more adjustments. [15:43:31] (03open) 10dancy: Bump docker-hub-mirror nginx memory requests/limits to 200Mi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/587 (https://phabricator.wikimedia.org/T424295) [15:43:31] (03update) 10dancy: Bump docker-hub-mirror nginx memory requests/limits to 200Mi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/587 (https://phabricator.wikimedia.org/T424295) [15:44:25] (03merge) 10dancy: Bump docker-hub-mirror nginx memory requests/limits to 200Mi [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/587 (https://phabricator.wikimedia.org/T424295) [15:55:41] 10GitLab, 06Release-Engineering-Team, 13Patch-For-Review: Gitlab CI broken - docker-hub-mirror.cloud.releng.team down? - https://phabricator.wikimedia.org/T424295#11856119 (10dancy) 05Open→03Resolved [16:08:51] (03open) 10dancy: test-requirements.txt: Use pytest>=9.0.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1137 (https://phabricator.wikimedia.org/T423568) [16:08:55] (03update) 10dancy: test-requirements.txt: Use pytest>=9.0.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1137 (https://phabricator.wikimedia.org/T423568) [16:13:43] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11856218 (10dancy) Thanks for the response @RLazarus >>! In T371069#11853857, @RLazarus wrote: > That said, here's the piece I'm not sure of, now that it... [16:19:40] (03update) 10dancy: test-requirements.txt: Use pytest>=9.0.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1137 (https://phabricator.wikimedia.org/T423568) [16:22:06] taavi I believe we'll be requesting LibUp for it yes. Before, on github, it was using depandabot [16:23:42] (03open) 10taavi: Add wikibase/new-lexeme-special-page [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/126 (https://phabricator.wikimedia.org/T424098) [16:23:46] (03update) 10taavi: Add wikibase/new-lexeme-special-page [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/126 (https://phabricator.wikimedia.org/T424098) [16:25:23] (03merge) 10dancy: test-requirements.txt: Use pytest>=9.0.3 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1137 (https://phabricator.wikimedia.org/T423568) [16:27:18] (03approved) 10wmde-leszek: Add wikibase/new-lexeme-special-page [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/126 (https://phabricator.wikimedia.org/T424098) (owner: 10taavi) [16:27:37] (03merge) 10taavi: Add wikibase/new-lexeme-special-page [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/126 (https://phabricator.wikimedia.org/T424098) [16:32:16] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-eventgate-4 in project deployment-prep - https://phabricator.wikimedia.org/T424253#11856278 (10bd808) `lang=shell-session $ ssh deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud The last Puppet run was at Fri... [16:33:19] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-eventgate-4 in project deployment-prep - https://phabricator.wikimedia.org/T424253#11856285 (10bd808) 05Open→03Invalid Error seems to have cleared on the https://prometheus-alerts.wmcloud.org/?q=%40state%3Dactive&... [16:37:27] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T424288#11856303 (10bd808) 05Open→03Resolved a:03bd808 Related to {T421244} `lang=shell-session bd808@deployment-puppetserver-1.deployment-prep.... [16:40:04] (03open) 10dancy: spiderpig: Make close button always work on job viewer page [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1138 [16:40:11] (03update) 10dancy: spiderpig: Make close button always work on job viewer page [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1138 [16:41:56] (03merge) 10dancy: spiderpig: Make close button always work on job viewer page [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1138 [16:42:44] (03open) 10dancy: Release 4.251.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1139 [16:42:50] 10GitLab (CI & Job Runners), 06collaboration-services, 07IPv6: IPv6 non-functional in GitLab CI environments - https://phabricator.wikimedia.org/T423310#11856314 (10CDobbins) How (assuming it's even possible) can we switch to the DigitalOcean runner? I tried adding the `cloud` tag to the `build` stage and ev... [16:45:08] (03merge) 10dancy: Release 4.251.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1139 [16:48:32] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [16:54:46] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MW-on-K8s: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11856365 (10RLazarus) Thanks for digging into it! I went back into IRC logs for more context (#mediawiki_security at 2024-07-26 00:00:21 UTC and beyond, i... [17:07:52] 10Continuous-Integration-Config, 07PHP 8.4 support: Update PHP 8.4 in CI to 8.4.10 - https://phabricator.wikimedia.org/T399131#11856390 (10Reedy) [17:10:12] 06Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MW-on-K8s, 06ServiceOps new: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11856399 (10RLazarus) [17:17:13] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-deploy05 in project deployment-prep - https://phabricator.wikimedia.org/T424115#11856421 (10bd808) 05Open→03Resolved a:03bd808 Resolved by deleting the instance as part of cleanup of {T421244}. [17:57:07] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep - https://phabricator.wikimedia.org/T424104#11856582 (10bd808) [17:57:10] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 06Discovery-Search (2026.03.03 - 2026.04.03): Migrate beta cluster to OpenSearch 2.x - https://phabricator.wikimedia.org/T421763#11856583 (10bd808) [18:14:06] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions: Adjust "Developer mode" checks in Phabricator Extensions - https://phabricator.wikimedia.org/T424377 (10Aklapper) 03NEW p:05Triage→03Low [18:14:10] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions: Adjust "Developer mode" checks in Phabricator Extensions - https://phabricator.wikimedia.org/T424377#11856667 (10Aklapper) [18:14:12] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11856668 (10Aklapper) [20:13:19] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T423877#11857043 (10thcipriani) p:05Triage→03Medium [20:14:06] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T423910#11857046 (10thcipriani) p:05Triage→03Medium a:03brennen [20:15:09] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T423911#11857051 (10thcipriani) p:05Triage→03Medium a:03Aklapper [20:15:48] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T423912#11857057 (10thcipriani) p:05Triage→03Medium a:03hashar [20:16:33] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T423913#11857063 (10thcipriani) p:05Triage→03Medium a:03dancy [20:17:38] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T423914#11857069 (10thcipriani) p:05Triage→03Medium a:03jnuche [20:18:10] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T423915#11857078 (10thcipriani) p:05Triage→03Medium a:03dduvall [21:29:24] (03update) 10dancy: backport: Only consider CI votes after backport approval [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1136 (https://phabricator.wikimedia.org/T420501) [21:29:59] thcipriani: for the deployment section of the DSS, it might be interesting for future editions to split by whether the responder has (mediawiki) deployment access or not [21:32:06] taavi: nice idea---we'd have to ask about it up-front, but I'll add it into our "next year" doc. What prompted that thought? [21:34:19] I had a similar thought reading the k8s deploys section. We meant to WMF prod k8s in that section, but I feel like some folks answered while thinking about toolforge. It was ambiguous though. [21:36:07] thcipriani: seeing the extremely high satisfaction numbers of spiderpig, and comparing that to things like https://wikitech.wikimedia.org/wiki/Talk:Deployments#Delete_the_UTC_morning_backport_window? is making me wonder if the prioritization in self-serviceability is having an unproportionally high negative impact on volunteers' and others without [21:36:07] possibility to get access ability to get stuff deployed [21:36:44] i.e. spiderpig is great if you're a foundation staffer regularly working on mediawiki that has ability to get deployment access by basically clicking 'request' [21:37:59] but if one comes from any other background and so can't just get deployment access that easily, the experience might be actually getting worse since knowledgeable folks regularly helping people get things shipped to production are not as often around during windows [21:39:51] that theory definitely tracks with https://www.mediawiki.org/wiki/Developer_Satisfaction_Survey/2026/Deployment#deployment-feedback [21:42:10] and, agreed, a big problem that SpiderPig did not make better. [21:43:18] jibes with the other big problem highlighted in the survey https://www.mediawiki.org/wiki/Developer_Satisfaction_Survey/2026/Code_review#feedback-on-process [21:44:27] the path to getting your code to prod without being someone who already gets your code to prod is hard to navigate. [21:57:50] (03update) 10dancy: backport: Only consider CI votes after backport approval [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1136 (https://phabricator.wikimedia.org/T420501) [21:59:35] (03update) 10dancy: backport: Only consider CI votes after backport approval [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1136 (https://phabricator.wikimedia.org/T420501) [22:02:36] (03update) 10dancy: backport: Only consider CI votes after backport approval [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1136 (https://phabricator.wikimedia.org/T420501) [22:43:09] (03update) 10thcipriani: backport: Only consider CI votes after backport approval [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1136 (https://phabricator.wikimedia.org/T420501) (owner: 10dancy) [22:43:10] (03approved) 10thcipriani: backport: Only consider CI votes after backport approval [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1136 (https://phabricator.wikimedia.org/T420501) (owner: 10dancy) [22:48:47] !log merged zuul3 branch of integration/config into master and pushed (in preparation for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277198) [22:48:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL