[00:03:48] !log Increase parallelism for wmf-beta-update-databases.py (T256168) [00:03:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:03:51] T256168: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168 [00:04:26] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, and 3 others: Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#11863752 (10bd808) The run logs are now exposed at https://beta-... [00:06:19] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cirrussearch12 on project deployment-prep - https://phabricator.wikimedia.org/T424104#11863753 (10bd808) a:03bking [01:08:32] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 10MW-on-K8s, 06ServiceOps new: Add helm rollback functionality to scap - https://phabricator.wikimedia.org/T371069#11863837 (10Scott_French) Very nice description of the problem statement in T371069#11853857 @RLazarus. Focusing on case 2 - i.e., an... [02:30:36] 10MediaWiki-Releasing, 05MW-1.46-release, 13Patch-For-Review: Update CREDITS in MW master for MW 1.46 - https://phabricator.wikimedia.org/T423258#11863963 (10ABreault-WMF) 05Openβ†’03Resolved a:03ABreault-WMF [02:34:46] (03update) 10mstyles: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) [02:37:25] (03update) 10mstyles: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) [02:41:58] 10MediaWiki-Releasing, 05MW-1.46-release, 13Patch-For-Review: Branch REL1_46 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T423259#11863991 (10ABreault-WMF) a:03ArielGlenn [02:44:45] (03update) 10mstyles: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) [02:47:22] 10MediaWiki-Releasing, 05MW-1.46-release, 13Patch-For-Review: Branch REL1_46 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T423259#11863995 (10ArielGlenn) 05Openβ†’03Resolved [02:51:03] (03update) 10mstyles: Scap: Add Security Deploy Command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1085 (https://phabricator.wikimedia.org/T361709) [03:08:18] note to folks that I am currently doing the mw 1.46 branch cut stuffs, along with arlo, it was nice to update the requirements to php 8.3 :-) [03:20:16] 10MediaWiki-Releasing, 05MW-1.46-release: Add REL1_46 to on-wiki documentation as the development snapshot - https://phabricator.wikimedia.org/T423261#11864014 (10ABreault-WMF) 05Openβ†’03Resolved a:03ArielGlenn https://www.mediawiki.org/w/index.php?title=Module%3AVersion&diff=8357805&oldid=8319522 [03:28:04] (and done) [04:37:52] (03PS4) 10Kevin Bazira: inference-services: trigger policy-violation image builds for requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/1271577 (https://phabricator.wikimedia.org/T418832) [07:50:34] (03CR) 10Hashar: "Can this be added to mediawiki/core instead? [includes/Composer/ComposerLaunchParallel.php](https://gerrit.wikimedia.org/g/mediawiki/core/" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1272025 (https://phabricator.wikimedia.org/T423059) (owner: 10Pwangai) [07:51:42] (03CR) 10Hashar: "We also used to have PHPUnit to generate Junit reports which have timing per test suite / tests etc. We dropped that because:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1272025 (https://phabricator.wikimedia.org/T423059) (owner: 10Pwangai) [09:22:30] 10MediaWiki-Releasing, 05MW-1.46-release: Send a branch announcement e-mail for MW REL1_46 - https://phabricator.wikimedia.org/T423260#11864657 (10MSantos) a:03MSantos [09:22:49] 10MediaWiki-Releasing, 05MW-1.46-release: Send a branch announcement e-mail for MW REL1_46 - https://phabricator.wikimedia.org/T423260#11864659 (10MSantos) Thank you @A_smart_kitten! [09:32:40] 10Fresh: xauth not installed on Fresh - https://phabricator.wikimedia.org/T424627 (10happy5214) 03NEW [09:32:50] 10MediaWiki-Releasing, 05MW-1.46-release: Send a branch announcement e-mail for MW REL1_46 - https://phabricator.wikimedia.org/T423260#11864743 (10MSantos) [09:33:11] 10MediaWiki-Releasing, 05MW-1.46-release: Send a branch announcement e-mail for MW REL1_46 - https://phabricator.wikimedia.org/T423260#11864748 (10MSantos) 05Openβ†’03Resolved [10:05:23] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create #Arabic-Sites tag - https://phabricator.wikimedia.org/T422567#11864974 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project #Arabic-Sites has been created: https://phabricator.wikimedia.org/project/view/8681/ (In case you... [10:09:08] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: wikibase-reconciliation - https://phabricator.wikimedia.org/T424637 (10Pintoch) 03NEW [10:40:26] !log sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud to fix failure seen in mwext-node24-rundoc #1717 [10:40:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:43:23] 06Project-Admins: Create a #traffic-rate-limit tag - https://phabricator.wikimedia.org/T424643 (10SLyngshede-WMF) 03NEW [10:43:53] 06Project-Admins: Create a #traffic-rate-limit tag - https://phabricator.wikimedia.org/T424643#11865142 (10SLyngshede-WMF) p:05Triageβ†’03Low I'll bring the topic up at next traffic team meeting. [10:53:46] 06Project-Admins: Create a #traffic-rate-limit tag - https://phabricator.wikimedia.org/T424643#11865196 (10Aklapper) 05Openβ†’03Stalled I'd say criteria how to recognize an issue being [not] in scope for this tag would also be welcome, to avoid mixups with other stuff going on like {T414805} or Thumbor 429s etc. [11:09:40] 10Continuous-Integration-Config, 07Code-Health: Could not open input file: tests/phpunit/generatePHPUnitConfig.php - https://phabricator.wikimedia.org/T424573#11865255 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF [11:12:40] 06Project-Admins: Create a #traffic-rate-limit tag - https://phabricator.wikimedia.org/T424643#11865286 (10A_smart_kitten) From my own personal perspective (as someone who occasionally does some Phab triaging), it'd also be good to know how/if this new tag should be used for tasks filed by end-users about rate-l... [11:35:57] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/EventLogging] Add EventBus phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1277673 (owner: 10Umherirrender) [11:37:44] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/EventLogging] Add EventBus phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1277673 (owner: 10Umherirrender) [11:58:42] (03open) 10phedenskog: Update metadata to server version [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/1 [12:32:59] Project mediawiki-core-doxygen build #20010: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/20010/ [12:36:50] (03open) 10phedenskog: Update from produuction. [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/2 [12:53:10] 10Gerrit, 06collaboration-services, 07Sustainability (Incident Followup): Fix rsync commands to avoid issues during switch of Gerrit instances - https://phabricator.wikimedia.org/T417137#11865687 (10MLechvien-WMF) Tagging #collaboration-services for triaging [12:59:51] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: Deploy Phab/Phorge 2026-04-28 - https://phabricator.wikimedia.org/T424656 (10brennen) 03NEW [13:00:36] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 07Essential-Work: Deploy Phab/Phorge 2026-04-28 - https://phabricator.wikimedia.org/T424656#11865758 (10Aklapper) a:03brennen [13:01:05] (03open) 10brennen: update phabricator submodule for 2026-04-28 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/102 (https://phabricator.wikimedia.org/T424656) [13:02:57] (03approved) 10aklapper: update phabricator submodule for 2026-04-28 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/102 (https://phabricator.wikimedia.org/T424656) (owner: 10brennen) [13:08:48] (03merge) 10brennen: update phabricator submodule for 2026-04-28 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/102 (https://phabricator.wikimedia.org/T424656) [13:11:01] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI [integration/config] - 10https://gerrit.wikimedia.org/r/1278450 [13:12:12] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11865824 (10ABran-WMF) I did not identify any obvious pattern in the recent build failures. Envoy logs the last http res... [13:12:43] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI [integration/config] - 10https://gerrit.wikimedia.org/r/1278450 (owner: 10Jforrester) [13:14:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI [integration/config] - 10https://gerrit.wikimedia.org/r/1278450 (owner: 10Jforrester) [13:14:59] !log Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI [13:15:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:15:56] 10Phabricator (Upstream), 07Upstream: Improve default parameters of default Maniphest saved search queries - https://phabricator.wikimedia.org/T187587#11865868 (10Aklapper) 05Openβ†’03Stalled > The current default setting for task search is status: open, group by: priority, sort by: priority, which is fairly... [13:16:02] 10Phabricator (Upstream), 07Upstream: Improve default parameters of default Maniphest global saved search queries - https://phabricator.wikimedia.org/T187587#11865873 (10Aklapper) [13:24:34] (03PS1) 10Jforrester: Zuul: [oojs/ui] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278459 [13:24:34] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278460 [13:24:34] (03PS1) 10Jforrester: Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278461 [13:24:34] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278462 [13:24:35] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278463 [13:24:38] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278464 [13:28:27] 06Release-Engineering-Team, 10Scap, 07Sustainability (Incident Followup): Improve scap/deploy rollback turnaround times - https://phabricator.wikimedia.org/T417166#11865943 (10MLechvien-WMF) More details about what went wrong in rollback are in the linked Incident document. cc @hnowlan as Incident Coordinato... [13:30:54] (03CR) 10Jforrester: [C:03+2] Zuul: [oojs/ui] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278459 (owner: 10Jforrester) [13:30:58] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278460 (owner: 10Jforrester) [13:32:32] (03Merged) 10jenkins-bot: Zuul: [oojs/ui] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278459 (owner: 10Jforrester) [13:32:35] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278460 (owner: 10Jforrester) [13:33:15] Yippee, build fixed! [13:33:15] Project mediawiki-core-doxygen build #20011: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/20011/ [13:34:40] !log Zuul: [oojs/ui] Drop PHP 8.2 testing [13:34:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:34:45] !log Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing [13:34:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:41:23] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: wikibase-reconciliation - https://phabricator.wikimedia.org/T424637#11866010 (10brennen) a:03brennen Sure, makes sense. One moment... [13:42:52] 10Gerrit, 06collaboration-services, 07Sustainability (Incident Followup): Fix rsync commands to avoid issues during switch of Gerrit instances - https://phabricator.wikimedia.org/T417137#11866016 (10ABran-WMF) 05Openβ†’03Resolved a:03ABran-WMF Thanks for the follow-up! This has been done already with... [13:47:25] 10GitLab (Project Migration), 06Release-Engineering-Team: Create new GitLab project group: wikibase-reconciliation - https://phabricator.wikimedia.org/T424637#11866037 (10brennen) 05Openβ†’03Resolved Created: https://gitlab.wikimedia.org/repos/wikibase-reconciliation Added Pintoch and DaxServer as owner... [13:51:56] (03open) 10brennen: phabricator submodule: followup fix to MFA on test instance [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/103 (https://phabricator.wikimedia.org/T424656) [13:52:15] (03merge) 10brennen: phabricator submodule: followup fix to MFA on test instance [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/103 (https://phabricator.wikimedia.org/T424656) [14:36:26] 06Release-Engineering-Team, 10MediaWiki-REST-API, 07Epic, 05MW-1.43-notes, and 2 others: [REST Sandbox] Remove SwaggerUI from MediaWiki Releases - https://phabricator.wikimedia.org/T397470#11866226 (10HCoplin-WMF) [14:53:29] (03PS5) 10Hashar: inference-services: trigger policy-violation image builds for requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/1271577 (https://phabricator.wikimedia.org/T418832) (owner: 10Kevin Bazira) [14:53:42] (03CR) 10Hashar: [C:03+2] inference-services: trigger policy-violation image builds for requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/1271577 (https://phabricator.wikimedia.org/T418832) (owner: 10Kevin Bazira) [14:55:32] (03Merged) 10jenkins-bot: inference-services: trigger policy-violation image builds for requirements.txt [integration/config] - 10https://gerrit.wikimedia.org/r/1271577 (https://phabricator.wikimedia.org/T418832) (owner: 10Kevin Bazira) [15:02:07] 10Phabricator (2026-04-28), 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 07Essential-Work: Deploy Phab/Phorge 2026-04-28 - https://phabricator.wikimedia.org/T424656#11866350 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d33eeb75-1fa0-412c-8124-2e1cbab4e7b1) set by j... [15:02:39] 10Phabricator (2026-04-28), 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 07Essential-Work: Deploy Phab/Phorge 2026-04-28 - https://phabricator.wikimedia.org/T424656#11866351 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cd773791-2be4-46c7-a171-c8391d0d2df5) set by j... [15:06:42] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1271577 (https://phabricator.wikimedia.org/T418832) (owner: 10Kevin Bazira) [15:09:43] (03CR) 10Pwangai: "It makes sense to move this logic to core, seems it will be more effective and impactful there. I will go ahead and close this patch." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1272025 (https://phabricator.wikimedia.org/T423059) (owner: 10Pwangai) [15:11:13] 10Phabricator (2026-04-28), 06Release-Engineering-Team (Doing 😎), 06collaboration-services, 07Essential-Work: Deploy Phab/Phorge 2026-04-28 - https://phabricator.wikimedia.org/T424656#11866401 (10Aklapper) 05Openβ†’03Resolved This deployment included https://gitlab.wikimedia.org/repos/phabricator/pha... [15:11:54] (03Abandoned) 10Pwangai: phpunit: log project breakdown and split-group timings [integration/quibble] - 10https://gerrit.wikimedia.org/r/1272025 (https://phabricator.wikimedia.org/T423059) (owner: 10Pwangai) [15:17:07] 10GitLab (Upstream pit of despair πŸ•³οΈ), 06Data-Engineering, 06Data-Engineering-Radar, 10DPE-Mediawiki-Content: Gitlab bug makes us have spurious artifact mismatch errors on YARN when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#11866444 (10xcollazo)... [15:25:33] Hmm, Phabricator feels a bit more sluggish to me. Might be related to the two changes we just merged which add DB queries [15:25:37] Curious if other folks have a similar experience (but I'm off soon to catch a flight) [15:31:45] 10MediaWiki-Releasing, 10ExtensionDistributor, 05MW-1.46-release: Add REL1_46 to ExtensionDistributor as the development snapshot - https://phabricator.wikimedia.org/T423262#11866549 (10MacFan4000) a:03MacFan4000 [15:37:37] (03open) 10trueg: wdqs-qlever: Docker iamge for the WDQS triple-store deployment. [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/166 (https://phabricator.wikimedia.org/T424670) [15:40:28] (03merge) 10dancy: wdqs-qlever: Docker iamge for the WDQS triple-store deployment. [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/166 (https://phabricator.wikimedia.org/T424670) (owner: 10trueg) [15:40:36] * A_smart_kitten wonders if there's a graph/data like https://grafana.wikimedia.org/goto/cfkgrmsb4w1kwb?orgId=1 but for Phab latency [15:42:40] 10MediaWiki-Releasing, 05MW-1.46-release: Send a branch announcement e-mail for MW REL1_46 - https://phabricator.wikimedia.org/T423260#11866629 (10ABreault-WMF) https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/WB47ECYPBNZBPI4MKTZFOPQI7WKDW4U4/ [15:43:46] (03approved) 10dduvall: .gitlab-ci.yml: Support [auto-deploy] for fast automated deployments [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/588 (owner: 10dancy) [15:56:01] (03merge) 10dancy: .gitlab-ci.yml: Support [auto-deploy] for fast automated deployments [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/588 [16:10:17] (03open) 10dancy: Spiderpig: Don't redraw job log on reconnect [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1145 [16:10:21] (03update) 10dancy: Spiderpig: Don't redraw job log on reconnect [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1145 [16:16:22] (03update) 10dancy: Spiderpig: Don't redraw job log on reconnect [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1145 [16:18:24] (03update) 10dancy: Spiderpig: Don't redraw job log on reconnect [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1145 [16:18:31] (03update) 10dancy: Spiderpig: Don't redraw job log on reconnect [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1145 [16:21:49] (03merge) 10dancy: Spiderpig: Don't redraw job log on reconnect [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1145 [16:21:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Skin-Architecture, 06Reader Experience Team, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI tests fail with TypeError: array_map(): Argument #2 ($array) must be of type array... - https://phabricator.wikimedia.org/T422907#11866921 [16:22:32] (03update) 10dancy: Release 4.254.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1146 [16:22:35] (03open) 10dancy: Release 4.254.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1146 [16:24:13] (03merge) 10dancy: Release 4.254.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1146 [16:25:35] (03CR) 10Kevin Bazira: "Thank you and welcome back! : )" [integration/config] - 10https://gerrit.wikimedia.org/r/1271577 (https://phabricator.wikimedia.org/T418832) (owner: 10Kevin Bazira) [16:28:33] FIRING: [2x] PuppetAgentFailure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:29:02] 10Phabricator, 06collaboration-services, 06Infrastructure-Foundations, 10Mail, 10VPS-project-Phabricator: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#11866948 (10A_smart_kitten) From [[https://we.phorge... [16:30:17] 10GitLab (Upstream pit of despair πŸ•³οΈ), 06Data-Engineering, 06Data-Engineering-Radar, 10DPE-Mediawiki-Content: Gitlab bug makes us have spurious artifact mismatch errors on YARN when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#11866954 (10amastilovi... [16:30:37] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278461 (owner: 10Jforrester) [16:30:39] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726#11866957 (10dancy) 05Openβ†’03Resolved Stopping at 1.35.1-do.3, the latest available today. [16:31:53] Project mediawiki-core-doxygen build #20015: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/20015/ [16:32:15] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278461 (owner: 10Jforrester) [16:32:24] 10Phabricator, 06MediaWiki-Platform-Team (Q3 Kanban Board): Silent bulk update for MediaWiki Platform - https://phabricator.wikimedia.org/T414778#11866987 (10A_smart_kitten) @OWresch-WMF @jtweed-wmf Given what's been said above, do you have any objection if the tasks in this hidden column are moved back to... [16:32:36] !log Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing [16:32:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:33:18] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Skin-Architecture, 06Reader Experience Team, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI tests fail with TypeError: array_map(): Argument #2 ($array) must be of type array... - https://phabricator.wikimedia.org/T422907#11866994 [16:33:30] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Skin-Architecture, 06Reader Experience Team, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI tests fail with TypeError: array_map(): Argument #2 ($array) must be of type array... - https://phabricator.wikimedia.org/T422907#11866995 [16:43:05] (03open) 10trueg: wdqs-streaming-consumer: Docker iamge for the WDQS streaming updater consumer. [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/167 (https://phabricator.wikimedia.org/T424679) [16:45:31] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Tmullapu] - https://phabricator.wikimedia.org/T424682 (10Tmullapu) 03NEW [16:52:00] 10GitLab (Upstream pit of despair πŸ•³οΈ), 10DPE-Mediawiki-Content, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Gitlab bug makes us have spurious artifact mismatch errors on YARN when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#11867072 (1... [16:52:10] 10GitLab (Upstream pit of despair πŸ•³οΈ), 10DPE-Mediawiki-Content, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st): Gitlab bug makes us have spurious artifact mismatch errors on YARN when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#11867074 (1... [17:04:05] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278462 (owner: 10Jforrester) [17:04:09] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278463 (owner: 10Jforrester) [17:04:14] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278464 (owner: 10Jforrester) [17:05:49] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278462 (owner: 10Jforrester) [17:05:51] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278463 (owner: 10Jforrester) [17:05:56] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1278464 (owner: 10Jforrester) [17:07:10] !log Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing [17:07:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:07:15] !log Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing [17:07:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:07:21] !log Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing [17:07:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:08:27] hey, just as a FYI for releng, it seems like there are currently two gitlab account approval requests that (fwics) have been open for ~17wks & ~12wks respectively, & that don't currently seem to be waiting for the filers to provide any additional requested information: T413600 & T416092 [17:08:28] T413600: Requesting GitLab account activation for Krrish Sehgal - https://phabricator.wikimedia.org/T413600 [17:08:28] T416092: Requesting GitLab account activation for xaoseric - https://phabricator.wikimedia.org/T416092 [17:32:57] Yippee, build fixed! [17:32:57] Project mediawiki-core-doxygen build #20016: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/20016/ [17:41:15] !log Unblock 79.192.0.0/10 [17:41:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:53:46] 10Phabricator, 06MediaWiki-Platform-Team (Q3 Kanban Board): Silent bulk update for MediaWiki Platform - https://phabricator.wikimedia.org/T414778#11867436 (10JTweed-WMF) Hi @A_smart_kitten no objection here. Thanks for sorting this. [18:04:55] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for tfmorris - https://phabricator.wikimedia.org/T424730 (10tfmorris) 03NEW [18:08:54] 10MediaWiki-Releasing, 10LibUp, 05MW-1.46-release: Configure LibraryUpgrader for REL1_46 - https://phabricator.wikimedia.org/T423263#11868046 (10Jgiannelos) a:03Jgiannelos [18:11:45] !log Unblock 86.0.0.0/16 [18:11:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:12:42] (03open) 10jgiannelos: Add REL1_46 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/127 [18:13:43] (03CR) 10Esanders: [C:03+1] "not that I'm aware of" [integration/config] - 10https://gerrit.wikimedia.org/r/1276782 (owner: 10Jforrester) [18:31:52] (03CR) 10Jforrester: [C:03+2] Zuul: [VisualEditor/VisualEditor] Replace Node 20 testing with Node 24 [integration/config] - 10https://gerrit.wikimedia.org/r/1276782 (owner: 10Jforrester) [18:52:15] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): PostBuild changing the status of successful builds to failure for no apparent reason - https://phabricator.wikimedia.org/T419488#11868322 (10SomeRandomDeveloper) I just had all... [19:03:38] (03PS2) 10Jforrester: Zuul: [VisualEditor/VisualEditor] Replace Node 20 testing with Node 24 [integration/config] - 10https://gerrit.wikimedia.org/r/1276782 [19:03:41] (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1276782 (owner: 10Jforrester) [19:13:01] (03approved) 10jforrester: Add REL1_46 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/127 (owner: 10jgiannelos) [19:13:05] (03merge) 10jforrester: Add REL1_46 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/127 (owner: 10jgiannelos) [19:13:25] (03merge) 10dancy: wdqs-streaming-consumer: Docker iamge for the WDQS streaming updater consumer. [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/167 (https://phabricator.wikimedia.org/T424679) (owner: 10trueg) [19:13:30] 10MediaWiki-Releasing, 10LibUp, 05MW-1.46-release, 13Patch-For-Review: Configure LibraryUpgrader for REL1_46 - https://phabricator.wikimedia.org/T423263#11868368 (10Jdforrester-WMF) 05Openβ†’03Resolved Thanks! [19:35:01] CI looks rather unhappy [19:35:13] Lots of jobs that aren't doing very much, or haven't for a while [19:35:25] well, castor specifically [19:35:32] >All nodes of label β€˜castor’ are offline [19:35:53] https://integration.wikimedia.org/ci/label/castor/ 1H45 since last success (and 1H48 for failure) [19:36:42] >Disk space is below threshold of 300.00 MiB. Only 20.00 KiB out of 313.91 GiB left on /srv/jenkins. [19:36:47] On integration-castor06 [19:37:36] 10Continuous-Integration-Infrastructure, 10Castor: integration-castor06 is basically out of disk space and blocking CI - https://phabricator.wikimedia.org/T424769 (10Reedy) 03NEW [19:37:44] 10Continuous-Integration-Infrastructure, 10Castor: integration-castor06 is basically out of disk space and blocking CI - https://phabricator.wikimedia.org/T424769#11868438 (10Reedy) p:05Triageβ†’03Unbreak! [19:41:39] all the castor-save-workspace-cache jobs are hung, which is preventing anything from merging. [19:42:13] cscott: see above :P [19:42:22] it's out of disk space [19:50:34] someone mentioned a few days ago that failing to save a cache should never cause the actual job to fail, but i guess that thought was never turned into code. [19:51:04] It's not failing... the jobs are just stalled [19:51:44] well, those jobs often fail as well, eg https://integration.wikimedia.org/ci/job/mediawiki-node24/10912/console [19:52:50] 10Continuous-Integration-Infrastructure, 10Castor: integration-castor06 is basically out of disk space and blocking CI - https://phabricator.wikimedia.org/T424769#11868484 (10Reedy) p:05Unbreak!β†’03High `lang=sh reedy@integration-castor06:/srv/castor/mediawiki-core/master$ df -h Filesystem Size Used A... [19:52:58] hey guys, you have 6GB back somehow [19:53:06] I was looking but did not delete anything [19:53:10] but things changed now [19:53:20] mutante: I just deleted some PHP81 caches in MW core master... [19:53:24] SchrΓΆdinger's df [19:53:31] Because we haven't run PHP 8.1 on master for a while [19:53:42] ACK. once I realized it's not / but /srv I did not know what can be deleted. [19:54:49] SchrΓΆdinger's cat /srv/castor/mediawiki-core/master [19:54:55] (03Merged) 10jenkins-bot: Zuul: [VisualEditor/VisualEditor] Replace Node 20 testing with Node 24 [integration/config] - 10https://gerrit.wikimedia.org/r/1276782 (owner: 10Jforrester) [20:10:00] ~24G free now [20:22:08] 10Phabricator, 06MediaWiki-Platform-Team (Q3 Kanban Board): Silent bulk update for MediaWiki Platform - https://phabricator.wikimedia.org/T414778#11868576 (10A_smart_kitten) {{done}} :] [20:26:37] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): integration-castor06 is basically out of disk space and blocking CI - https://phabricator.wikimedia.org/T424769#11868597 (10A_smart_kitten) [20:29:22] 10MediaWiki-Releasing, 10ExtensionDistributor, 05MW-1.46-release, 13Patch-For-Review: Add REL1_46 to ExtensionDistributor as the development snapshot - https://phabricator.wikimedia.org/T423262#11868603 (10Jdforrester-WMF) 05Openβ†’03Resolved Deployed. [20:31:20] (03update) 10thcipriani: git.py: info(): Use head_sha1 for head key [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1144 (owner: 10dancy) [20:31:56] thanks A_smart_kitten, i'll have a look. [20:32:23] (re: gitlab account reqs from a while ago.) [20:33:59] (03merge) 10thcipriani: git.py: info(): Use head_sha1 for head key [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1144 (owner: 10dancy) [21:01:51] (03update) 10dancy: Release 4.255.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1147 [21:01:54] (03open) 10dancy: Release 4.255.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1147 [21:03:39] (03merge) 10dancy: Release 4.255.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1147 [21:15:41] 10MediaWiki-Releasing, 05MW-1.46-release, 05Release: Release MediaWiki 1.46.0 - https://phabricator.wikimedia.org/T423255#11868727 (10cscott) [21:15:42] James_F: (not urgent!) how do all these https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1278592 (bump mw requirement version) get generated? should $someone in the mw release group know how to do this? [21:24:01] apergos: I've got a terrible Python script that I keep meaning to clean up and submit as a tool other people can use. Yeah, I should do the work, maybe during the Hackathon. [21:24:21] <3 [21:24:52] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06collaboration-services, 06Traffic: Purge frontend cache when publish new coverage report under https://doc.wikimedia.org/cover - https://phabricator.wikimedia.org/T423951#11868760 (10Dzahn) @Umherirrender The maximum time to wait i... [21:24:57] even the awful script should live somewhere, even if you need a huge README caveat "this needs all kinds of tweaks". [21:25:10] or "kids, do not run this at home" :-D [21:26:45] Yes, but right now it just has my credentials hard-coded into it, so… :-) [21:26:55] (03PS1) 10Jforrester: Zuul: Drop experimental node22 jobs, never used in practice [integration/config] - 10https://gerrit.wikimedia.org/r/1278615 [21:26:56] (03PS1) 10Jforrester: jjb: Drop unused node20 and node22 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1278616 [21:26:56] (03PS1) 10Jforrester: Docker: Drop node22-test-browser-php83-composer, unused [integration/config] - 10https://gerrit.wikimedia.org/r/1278617 [21:26:56] (03PS1) 10Jforrester: jjb: Provide api-testing-{mysql,sqlite}-php85 [integration/config] - 10https://gerrit.wikimedia.org/r/1278618 [21:26:57] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/api-testing] Add PHP 8.5 experimentals [integration/config] - 10https://gerrit.wikimedia.org/r/1278619 [21:26:59] (03PS1) 10Jforrester: jjb: Drop a handful of unused PHP 8.1 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1278620 [21:32:36] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 05MW-1.46-release: Prune /vendor for REL1_46 - https://phabricator.wikimedia.org/T423266#11868781 (10ArielGlenn) a:03ArielGlenn [21:41:09] cscott: the castor should not block thing is T423557 [21:41:13] T423557: Realistic measure castor wait times and make to decrease the time - https://phabricator.wikimedia.org/T423557 [22:01:52] (03PS3) 10Jeena Huneidi: WikiLambda selenium: Use ci-client image [integration/config] - 10https://gerrit.wikimedia.org/r/1277759 (https://phabricator.wikimedia.org/T423330) [22:02:40] (03PS4) 10Jeena Huneidi: WikiLambda selenium: Use ci-client image [integration/config] - 10https://gerrit.wikimedia.org/r/1277759 (https://phabricator.wikimedia.org/T423330) [22:08:52] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): integration-castor06 is basically out of disk space and blocking CI - https://phabricator.wikimedia.org/T424769#11868876 (10Reedy) ` --- /srv/castor ---------------------------------------------------------------... [22:43:56] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 05MW-1.46-release, 13Patch-For-Review: Prune /vendor for REL1_46 - https://phabricator.wikimedia.org/T423266#11868993 (10ArielGlenn) Output from composer for this run: ` Package operations: 3 installs, 14 updates, 50 removals Installs: composer/installers:v2.3.0,... [22:47:39] 10Continuous-Integration-Infrastructure, 10Castor, 07ci-test-error (WMF-deployed Build Failure): integration-castor06 is basically out of disk space and blocking CI - https://phabricator.wikimedia.org/T424769#11869003 (10Reedy) Deleted some more older PHP version and node20 folders ` --- /srv/castor -------... [22:58:46] 06Release-Engineering-Team (Doing 😎), 03Wikimedia-Hackathon-2026: How MediaWiki code gets deployed, things we are changing, and Q&A - https://phabricator.wikimedia.org/T424780 (10bd808) 03NEW [23:03:11] 06Release-Engineering-Team (Doing 😎), 03Wikimedia-Hackathon-2026: How MediaWiki code gets deployed and things we are changing - https://phabricator.wikimedia.org/T424780#11869068 (10bd808) [23:11:09] 06Release-Engineering-Team (Doing 😎), 03Wikimedia-Hackathon-2026: How MediaWiki code gets deployed and things we are changing - https://phabricator.wikimedia.org/T424780#11869093 (10bd808) [23:28:33] 06Release-Engineering-Team (Doing 😎), 03Wikimedia-Hackathon-2026: How MediaWiki code gets deployed and things we are changing - https://phabricator.wikimedia.org/T424780#11869141 (10bd808) Questions for @bd808 to remember to ask folks who attend the session: How do you reason about where your code is in the d... [23:29:34] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-eventgate-4 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:29:43] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-eventgate-4 on project deployment-prep - https://phabricator.wikimedia.org/T424784 (10wmcs-alerts) 03NEW [23:30:30] 10MediaWiki-Releasing, 10MediaWiki-Vendor, 05MW-1.46-release, 13Patch-For-Review: Prune /vendor for REL1_46 - https://phabricator.wikimedia.org/T423266#11869156 (10ArielGlenn)