[01:14:45] 06Release-Engineering-Team, 10Codex: Codex origin/main issue - https://phabricator.wikimedia.org/T423066 (10Volker_E) 03NEW [03:08:38] seems like zuul is broken, nothing happens when I click +2 [03:13:54] or maybe gerrit is broken [03:15:04] Yeah, I +2'd https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1268278 ~4 hours ago, and no gate-and-submit has kicked off. [03:16:01] Hm, and that comment I just left on it hasn't shown up in the normal IRC-notifications. Something has definitely gotten disconnected. [03:16:57] I think it's more likely gerrit than zuul [03:18:07] I'd imagine it's aftereffects of T423027, even though Gerrit itself is back up. [03:18:08] T423027: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027 [03:19:52] Caused by: java.io.IOException: No space left on device [03:26:12] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11812527 (10tstarling) Here's the Gerrit error log when I hit +2 on a core change. Lucene throws an exception with "No space left on device". ` [2026-04-13T03:23:16... [03:31:40] going to restart gerrit [03:33:02] Project beta-code-update-eqiad build #595861: 04FAILURE in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/595861/ [03:35:42] ok, that fixed it, gate jobs now running [03:38:29] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11812529 (10tstarling) Presumably Lucene got that "no space" exception and closed the writer, storing the exception. Then every time it tried to write to the index,... [03:45:18] Yippee, build fixed! [03:45:18] Project beta-code-update-eqiad build #595862: 09FIXED in 2 min 18 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/595862/ [04:17:28] 06Release-Engineering-Team, 10Codex: Codex origin/main issue - https://phabricator.wikimedia.org/T423066#11812574 (10Volker_E) 05Open→03Invalid After several hours of unsuccessful rebasing, then writing this task and then another hour of repeating trying to fix the issue, I gladly was able to fix this... [04:54:47] 10Phabricator, 06Infrastructure-Foundations, 10Mail: Mails do not arrive any more - https://phabricator.wikimedia.org/T423055#11812591 (10Aklapper) 05Open→03Stalled You could check/search on https://phabricator.wikimedia.org/mail/. (I have not experienced a problem with my Phabricator mail notifications... [06:06:53] 10Phabricator, 06Infrastructure-Foundations, 10Mail: Mails do not arrive any more - https://phabricator.wikimedia.org/T423055#11812685 (10IKhitron) >>! In T423055#11812591, @Aklapper wrote: > You could check/search on https://phabricator.wikimedia.org/mail/. > (I have not experienced a problem with my Phabri... [06:10:58] 10Phabricator, 06Infrastructure-Foundations, 10Mail: Mails do not arrive any more - https://phabricator.wikimedia.org/T423055#11812686 (10Aklapper) 05Stalled→03Invalid Glad it works again for you! [06:13:38] 10Phabricator, 06Infrastructure-Foundations, 10Mail: Mails do not arrive any more - https://phabricator.wikimedia.org/T423055#11812689 (10IKhitron) Thank you. [06:15:22] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11812690 (10IKhitron) Does this mean I can try to recheck once again now? [06:18:37] 10GitLab, 06Release-Engineering-Team (Doing 😎), 07Essential-Work: GitLab Private Repository Request for: WME exceptions - https://phabricator.wikimedia.org/T422909#11812691 (10KCVelaga_WMF) Hi @brennen, thanks for creating. I think I am not yet added to the repo, I am getting a 404 error when trying to a... [06:51:14] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11812709 (10tstarling) >>! In T423027#11812690, @IKhitron wrote: > Does this mean I can try to recheck once again now? Yes. [06:58:48] (03PS1) 10Phedenskog: mysql: Disable the redo log for every commit [integration/quibble] - 10https://gerrit.wikimedia.org/r/1270277 (https://phabricator.wikimedia.org/T423023) [07:03:08] (03CR) 10CI reject: [V:04-1] mysql: Disable the redo log for every commit [integration/quibble] - 10https://gerrit.wikimedia.org/r/1270277 (https://phabricator.wikimedia.org/T423023) (owner: 10Phedenskog) [07:05:15] (03PS1) 10Phedenskog: postgresql: Async write changes to the write ahead log [integration/quibble] - 10https://gerrit.wikimedia.org/r/1270278 (https://phabricator.wikimedia.org/T423070) [07:12:04] (03PS1) 10Phedenskog: sqlite: Disable flushing per disk per commit [integration/quibble] - 10https://gerrit.wikimedia.org/r/1270280 (https://phabricator.wikimedia.org/T423070) [07:12:35] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2003:443) - https://phabricator.wikimedia.org/T423034#11812750 (10Jelto) [07:14:04] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit2003:443) - https://phabricator.wikimedia.org/T423034#11812753 (10Jelto) 05Open→03Resolved a:03Jelto This issue is tracked in T423027, I'll close this task. [07:19:48] (03PS1) 10Phedenskog: mysql: Use persistent connections [integration/quibble] - 10https://gerrit.wikimedia.org/r/1270283 (https://phabricator.wikimedia.org/T423070) [08:10:24] 10Phabricator, 06MediaWiki-Platform-Team (Q3 Kanban Board): Silent bulk update for MediaWiki Platform - https://phabricator.wikimedia.org/T414778#11812906 (10A_smart_kitten) Thanks for the replies @OWresch-WMF @JTweed-WMF :) For context, it seems like (at least) some of the tasks currently in that hidden... [08:19:03] 10Phabricator: Username not updated after global rename - https://phabricator.wikimedia.org/T423007#11812944 (10Sandra_Hanbo) Hello, Yes, on Phabricator, I would like it to be as my new MediaWiki name: SanBonne, and not my full name [08:20:23] 10Phabricator: Username not updated after global rename - https://phabricator.wikimedia.org/T423007#11812949 (10A_smart_kitten) 05Invalid→03Open (Reopening per @sandra_hanbo's last comment; [[https://phabricator.wikimedia.org/maniphest/query/T7cDG0kbjIUo/#R|IIUC]] tasks can also be filed on Phab to request P... [08:20:45] 10Phabricator: Rename Phabricator account to SanBonne - https://phabricator.wikimedia.org/T423007#11812951 (10A_smart_kitten) [08:39:15] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Rename Phabricator account to SanBonne - https://phabricator.wikimedia.org/T423007#11813014 (10Aklapper) 05Open→03Resolved a:03Aklapper Done. [08:40:12] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Rename Phabricator account to SanBonne - https://phabricator.wikimedia.org/T423007#11813046 (10SanBonne) Thanks [08:44:40] 06Release-Engineering-Team: kokkuri:build-and-run-image should detect variants with no entrypoint and fail gracefully - https://phabricator.wikimedia.org/T423087 (10gmodena) 03NEW [09:10:41] 06Release-Engineering-Team, 06Infrastructure-Foundations: Add tox-uv support to the tox-v{3,4} Docker images - https://phabricator.wikimedia.org/T421348#11813380 (10elukey) @hashar ping :) [09:12:52] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11813409 (10Jelto) a:03Jelto Thank you to everyone who troubleshooted and mitigated this issue on a weekend. > on gerrit2003 restarted gerrit T42302... [09:20:31] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11813421 (10Jelto) [09:27:40] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11813473 (10Jelto) This has caused disk space issues again in T423027. This task is already tagged with #wikimedia-inciden... [09:31:50] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11813499 (10IKhitron) >>! In T423027#11812709, @tstarling wrote: >>>! In T423027#11812690, @IKhitron wrote: >> Does this mean I can try to recheck once again now? >... [09:41:16] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11813524 (10hnowlan) Could I be added to the project admins ACL please? I would like to add tags to manage incident severity in our incident response and follo... [09:50:34] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11813543 (10A_smart_kitten) [11:56:44] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#11813968 (10Ladsgroup) >>! In T706#11813524, @hnowlan wrote: > Could I be added to the project admins ACL please? I would like to add tags to manage incident s... [12:37:43] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services: Alert when Gerrit CI (Zuul, Jenkins, Gearman) is down/stuck - https://phabricator.wikimedia.org/T423123 (10Jelto) 03NEW [12:38:04] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Alert when Gerrit CI (Zuul, Jenkins, Gearman) is down/stuck - https://phabricator.wikimedia.org/T423123#11814086 (10Jelto) [12:38:24] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11814090 (10Jelto) [12:42:30] (03PS2) 10Aghirelli: phan: Add FlaggedRevs dependency for WikimediaCustomizations [integration/config] - 10https://gerrit.wikimedia.org/r/1270062 (https://phabricator.wikimedia.org/T421011) [13:43:36] (03CR) 10A smart kitten: phan: Add FlaggedRevs dependency for WikimediaCustomizations (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1270062 (https://phabricator.wikimedia.org/T421011) (owner: 10Aghirelli) [14:19:42] (03PS3) 10Aghirelli: zuul: Add FlaggedRevs dependency for WikimediaCustomizations [integration/config] - 10https://gerrit.wikimedia.org/r/1270062 (https://phabricator.wikimedia.org/T421011) [14:19:50] (03CR) 10Aghirelli: zuul: Add FlaggedRevs dependency for WikimediaCustomizations (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1270062 (https://phabricator.wikimedia.org/T421011) (owner: 10Aghirelli) [14:39:23] (03PS4) 10Jforrester: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep [integration/config] - 10https://gerrit.wikimedia.org/r/1270062 (https://phabricator.wikimedia.org/T421011) (owner: 10Aghirelli) [14:40:05] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep [integration/config] - 10https://gerrit.wikimedia.org/r/1270062 (https://phabricator.wikimedia.org/T421011) (owner: 10Aghirelli) [14:41:58] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep [integration/config] - 10https://gerrit.wikimedia.org/r/1270062 (https://phabricator.wikimedia.org/T421011) (owner: 10Aghirelli) [14:42:16] !log Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep, for T421011 [14:42:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:42:19] T421011: Implement support for FlaggedRevs in citation count for Attribution API - https://phabricator.wikimedia.org/T421011 [15:16:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11815011 (10LSobanski) p:05Medium→03High [15:20:21] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11815038 (10jnuche) >>! In T418521#11807710, @jnuche wrote: > @Dzahn Would it make sense to wait until @has... [15:50:51] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Alert when Gerrit CI (Zuul, Jenkins, Gearman) is down/stuck - https://phabricator.wikimedia.org/T423123#11815234 (10LSobanski) p:05Triage→03High [15:57:27] 10GitLab, 06Release-Engineering-Team (Doing 😎), 07Essential-Work: GitLab Private Repository Request for: WME exceptions - https://phabricator.wikimedia.org/T422909#11815270 (10brennen) Yep, I mistakenly assumed you'd be a member of the higher-level `repos/wme` group. Sent you an invite just now. [16:16:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11815369 (10Dzahn) Well, I think it was about actually doing the switch but after the latest comments I rev... [18:28:43] 10Continuous-Integration-Config, 06Community-Tech, 10MediaWiki-extensions-CommunityRequests: CI failure for CommunityRequests: Missing dependency on TemplateData? - https://phabricator.wikimedia.org/T423186 (10Jdforrester-WMF) 03NEW [18:34:54] 10Continuous-Integration-Config, 06Community-Tech, 10MediaWiki-extensions-CommunityRequests: CI failure for CommunityRequests: Missing dependency on TemplateData? - https://phabricator.wikimedia.org/T423186#11816426 (10Jdforrester-WMF) →14Duplicate dup:03T401638 [20:05:47] 06Release-Engineering-Team (Doing 😎), 07Epic: [Epic] Migrate gitlab-cloud-runner to WMCS - https://phabricator.wikimedia.org/T416256#11816733 (10Andrew) I owe y'all an update about Magnum! The news is not ideal: when Magnum leadership deprecated the Heat driver they directed deployers to adopt one of two CAPI... [20:19:39] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Skin-Architecture, 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: CI tests fail with TypeError: array_map(): Argument #2 ($array) must be of type array, null given - https://phabricator.wikimedia.org/T422907#11816783 (10Jdlrobson-W... [20:23:22] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, unconfigured in repo [integration/config] - 10https://gerrit.wikimedia.org/r/1270553 [20:23:30] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, unconfigured in repo [integration/config] - 10https://gerrit.wikimedia.org/r/1270553 (owner: 10Jforrester) [20:25:06] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, unconfigured in repo [integration/config] - 10https://gerrit.wikimedia.org/r/1270553 (owner: 10Jforrester) [20:25:44] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used [integration/config] - 10https://gerrit.wikimedia.org/r/1270553 [20:25:47] (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1270553 (owner: 10Jforrester) [20:25:48] (03CR) 10jenkins-bot: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used [integration/config] - 10https://gerrit.wikimedia.org/r/1270553 (owner: 10Jforrester) [20:28:09] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used [integration/config] - 10https://gerrit.wikimedia.org/r/1270553 (owner: 10Jforrester) [20:28:29] !log Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used [20:28:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:51:48] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11816922 (10Dzahn) **update April 2026** | SPARE | gerrit2002.wikimedia.org | gerrit-spare.wikimedia.org || | Partition... [20:53:38] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T394858) [20:53:41] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T394858) [20:55:17] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T394858) [20:55:29] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T394858) [21:04:21] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T394858) [21:23:54] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11817082 (10Dzahn) Suggested run book to move remaining data: prep: - mkdir /srv/gerrit/site_path (we need/should use... [21:34:51] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11817117 (10Dzahn) [21:39:25] 10Gerrit, 06collaboration-services, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11817132 (10Dzahn) We had this existing old ticket to move all Gerrit data out of the root partition; where a huge chunk of LFS data had already been moved but not a... [21:45:27] (03PS1) 10Jforrester: Zuul: Re-write growth-experiments-selenium-gate to not add jobs on release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1270576 [21:46:20] 06Project-Admins, 06tech-decision-forum: Archive tech-decision-forum - https://phabricator.wikimedia.org/T368599#11817145 (10A_smart_kitten) 05Declined→03Open (boldly reopening per my previous comment) [21:47:06] (03CR) 10CI reject: [V:04-1] Zuul: Re-write growth-experiments-selenium-gate to not add jobs on release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1270576 (owner: 10Jforrester) [21:53:25] (03CR) 10Jforrester: Use standalone jobs for Wikibase Selenium tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/676107 (https://phabricator.wikimedia.org/T287582) (owner: 10Hashar) [21:53:29] (03CR) 10Jforrester: Zuul: use standalone jobs for GrowthExperiments Cypress tests (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1268559 (https://phabricator.wikimedia.org/T417412) (owner: 10Zfilipin) [21:56:07] (03PS2) 10Jforrester: Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches [integration/config] - 10https://gerrit.wikimedia.org/r/1270576 [21:58:18] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches [integration/config] - 10https://gerrit.wikimedia.org/r/1270576 (owner: 10Jforrester) [21:59:55] (03Merged) 10jenkins-bot: Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches [integration/config] - 10https://gerrit.wikimedia.org/r/1270576 (owner: 10Jforrester) [22:00:18] !log Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches [22:00:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:00:37] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11817190 (10Dzahn) Created a new ed25519 key pair for new zuul to connect to Gerrit (in the future). ` [Ops] [puppet-private] (333... [22:47:42] 10Phabricator-Bot-Requests: Requesting a Phabricator bot account for a QS dashboard - https://phabricator.wikimedia.org/T423209 (10SLong-WMF) 03NEW [22:57:54] are code coverage tests broken? [22:57:55] https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-patch/68337/console [23:54:47] bawolff: Eurgh. They were working last week…