[00:10:15] 10Phabricator, 07Documentation: Batch edit silencing instructions seem to be missing some information - https://phabricator.wikimedia.org/T423526#11832164 (10bd808) >>! In T423526#11830894, @Aklapper wrote: > I think I'd rather not change that, unless you have strong opinions. Probably not strong, no. The `ph... [00:14:29] mutante: I haven't gotten the work done to replace the Beta Cluster deploy server with a bookworm instance yet (T421244). I told you I thought I would be done by my end of day today. Will you be blocked if I don't get it done until sometime on Monday? [00:14:30] T421244: Replace deployment-deploy04 with a Bookworm instance with Java 21 - https://phabricator.wikimedia.org/T421244 [00:15:41] thcipriani: ^ same question I guess. How bad would it screw up the world for your hopes and dreams if I walk away until Monday? [00:17:08] bd808: do Monday! no worries [00:17:45] bd808: ^ the world for my hopes and dreams was already screwed up [00:17:59] y'all are the coolest. [00:27:46] ice cold [05:58:18] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#11832487 (10LSobanski) @bd808 In addition to the good point raised by @A_smart_kitten above the general intent here is to reduce complexity. Leaving a dependen... [07:05:07] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review, 07User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#11832582 (10Nemoralis) [08:18:40] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, and 2 others: gerrit: Adapt timeouts to avoid 502 errors in CI jobs - https://phabricator.wikimedia.org/T421827#11832698 (10ABran-WMF) I have tried to limit `max_concurrent_streams` to 50, still inconclusive for the connection inter... [09:02:27] 10Gerrit, 06collaboration-services, 07Documentation, 07Sustainability (Incident Followup): Update and improve operation runbooks and documentation for Gerrit - https://phabricator.wikimedia.org/T423601#11832752 (10ABran-WMF) [09:02:28] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07Sustainability (Incident Followup): Alert when Gerrit CI (Zuul, Jenkins, Gearman) is down/stuck - https://phabricator.wikimedia.org/T423123#11832753 (10ABran-WMF) [09:02:29] 10Gerrit, 06collaboration-services, 13Patch-For-Review, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11832751 (10ABran-WMF) [09:03:08] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11832756 (10ABran-WMF) [09:03:09] 10Gerrit, 06collaboration-services, 13Patch-For-Review, 07Wikimedia-Incident: 2026-04-12 Gerrit Outage (was: DiskSpace) - https://phabricator.wikimedia.org/T423027#11832755 (10ABran-WMF) [09:05:14] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review, 07Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#11832759 (10ABran-WMF) [09:12:14] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11832767 (10jnuche) @Dzahn I restarted Jenkins on contint1003 so the new plugins would get picked up. Now both... [10:47:04] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: New base images without mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423622#11833147 (10MoritzMuehlenhoff) >>! In T423622#11830895, @thcipriani wrote: > Updated task description to clarify, yes, the S... [10:47:15] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: New base images without mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423622#11833149 (10MoritzMuehlenhoff) p:05Triage→03High [12:58:15] 06Project-Admins, 07Gender-Support, 07I18n, 07RTL: Duplication: "RTL" and "Gender" are both Phab project tags and also workboard columns of the #i18n project tag - https://phabricator.wikimedia.org/T238953#11833427 (10Aklapper) 05Open→03Declined [13:04:29] 10Phabricator (Upstream), 07Upstream: Provide an easy way to link to a Phabricator task in a user-friendly way - https://phabricator.wikimedia.org/T388243#11833446 (10Aklapper) 05Open→03Declined I'm going to paste the same comment here which I posted in upstream: > I believe copying the URL is easiest... [13:06:33] 06Project-Admins, 10MediaWiki-Special-pages, 06Product Safety and Integrity: Split user-muting features (incl. Special:Mute) into its own Phabricator project (e.g. MediaWiki-Mute? / MediaWiki-Special-Mute?) - https://phabricator.wikimedia.org/T409674#11833453 (10Aklapper) [13:43:55] 10Gerrit, 06collaboration-services, 07Documentation, 07Sustainability (Incident Followup): Update and improve operation runbooks and documentation for Gerrit - https://phabricator.wikimedia.org/T423601#11833558 (10ABran-WMF) I've added a bit of documentation on gerrit ssh commands: https://wikitech.wikimed... [13:46:29] 10Gerrit, 06collaboration-services, 07Documentation, 07Sustainability (Incident Followup): Update and improve operation runbooks and documentation for Gerrit - https://phabricator.wikimedia.org/T423601#11833563 (10ABran-WMF) I've also updated the "restart gerrit" part: https://wikitech.wikimedia.org/wiki/G... [13:47:40] 10Gerrit, 06collaboration-services, 07Documentation, 07Sustainability (Incident Followup): Update and improve operation runbooks and documentation for Gerrit - https://phabricator.wikimedia.org/T423601#11833567 (10ABran-WMF) [14:19:09] (03PS2) 10SBassett: Use allow-listed User Agent for fresh gerrit downloads [fresh] - 10https://gerrit.wikimedia.org/r/1272800 (https://phabricator.wikimedia.org/T421726) [14:19:30] (03Abandoned) 10SBassett: Use allow-listed User Agent for fresh gerrit downloads [fresh] - 10https://gerrit.wikimedia.org/r/1272800 (https://phabricator.wikimedia.org/T421726) (owner: 10SBassett) [14:34:59] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Plovdiv 25): Investigate if there's a way to make castor wait time smaller - https://phabricator.wikimedia.org/T423557#11833713 (10Peter) @bd808 yes I like that idea. I'm thinking there potentially two things that can... [14:53:33] 10Fresh, 10Gerrit, 06collaboration-services: Wikimedia gerrit load management 429s break fresh-install - https://phabricator.wikimedia.org/T421726#11833797 (10sbassett) The patch from above won't work, after chatting with @ABran-WMF. Currently exploring other options with them/SRE. [15:18:58] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T394858 https://phabricator.wikimedia.org/T396106) [15:19:34] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T394858 https://phabricator.wikimedia.org/T396106) [15:26:44] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T390531 https://phabricator.wikimedia.org/T394858 https://phabricator.wikimedia.org/T396106) [15:26:49] (03update) 10dancy: sync-world: Offer to rollback k8s deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1129 (https://phabricator.wikimedia.org/T225207 https://phabricator.wikimedia.org/T375497 https://phabricator.wikimedia.org/T390531 https://phabricator.wikimedia.org/T394858 https://phabricator.wikimedia.org/T396106) [15:27:57] 06Release-Engineering-Team, 10Scap, 06serviceops-deprecated, 06SRE-OnFire, 07Sustainability (Incident Followup): Should scap be able to update helmfile-defaults when -Dbuild_mw_container_image:False ? - https://phabricator.wikimedia.org/T390531#11833922 (10dancy) >>! In T390531#11831389, @dancy wrote... [16:42:57] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726 (10dancy) 03NEW [16:43:44] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726#11834255 (10dancy) [16:46:27] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726#11834269 (10dancy) [16:49:33] !log Upgrading gitlab cloud runners (staging) k8s from 1.32.10-do.1 to 1.32.13-do.2 (T423726) [16:49:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:49:37] T423726: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726 [16:59:47] 06Release-Engineering-Team, 06collaboration-services: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#11834309 (10Dzahn) The same should happen with phab2003 replacing phab2002 over at T423727. [17:10:12] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Upgrade K3s cluster to most recent stable version - https://phabricator.wikimedia.org/T400077#11834361 (10jnuche) 05Open→03Resolved Production cluster is now running `v1.35.3+k3s1` [17:19:59] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T420482#11834404 (10thcipriani) 05Open→03Resolved [17:24:19] (03open) 10dancy: README.md: Update cluster upgrade notes [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/568 [17:24:22] (03update) 10dancy: README.md: Update cluster upgrade notes [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/568 [17:43:48] (03update) 10dancy: README.md: Update cluster upgrade notes [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/568 [17:43:51] (03update) 10dancy: README.md: Update cluster upgrade notes [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/568 [17:45:05] (03merge) 10dancy: README.md: Update cluster upgrade notes [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/568 [17:46:24] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726#11834487 (10dancy) [17:47:42] !log Upgrading gitlab cloud runners (prod) k8s from 1.32.10-do.1 to 1.32.13-do.2 (T423726) [17:47:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:47:44] T423726: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726 [17:49:40] (03open) 10dancy: staging/digitalocean.tfvars: kubernetes_version = "1.32.13-do.2" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/569 (https://phabricator.wikimedia.org/T423726) [17:49:41] (03update) 10dancy: staging/digitalocean.tfvars: kubernetes_version = "1.32.13-do.2" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/569 (https://phabricator.wikimedia.org/T423726) [17:50:21] (03merge) 10dancy: staging/digitalocean.tfvars: kubernetes_version = "1.32.13-do.2" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/569 (https://phabricator.wikimedia.org/T423726) [18:12:00] (03open) 10dancy: production/digitalocean.tfvars: kubernetes_version = "1.32.13-do.2" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/570 (https://phabricator.wikimedia.org/T423726) [18:12:02] (03update) 10dancy: production/digitalocean.tfvars: kubernetes_version = "1.32.13-do.2" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/570 (https://phabricator.wikimedia.org/T423726) [18:15:50] castor-save-workspace-cache is failing pretty often [18:16:40] Not sure if this is known [18:56:24] indeed it is, failing a lot on my fix for T282893. Are you seeing it break builds, Dreamy_Jazz ? [18:56:24] T282893: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893 [18:57:59] It like failed three times in a row on a patch I was looking to merge [18:58:07] link? [18:58:31] want to be sure I'm looking at the right builds [18:58:49] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/1261502 [18:58:53] It has now merged [18:58:59] thanks [18:59:06] But in the test stage was failing [19:15:06] (03update) 10dancy: production/digitalocean.tfvars: kubernetes_version = "1.32.13-do.2" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/570 (https://phabricator.wikimedia.org/T423726) [19:16:24] (03merge) 10dancy: production/digitalocean.tfvars: kubernetes_version = "1.32.13-do.2" [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/570 (https://phabricator.wikimedia.org/T423726) [19:21:17] (03open) 10dancy: istio/main.tf: Bump Istio to version 1.29.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/571 (https://phabricator.wikimedia.org/T423726) [19:21:18] (03update) 10dancy: istio/main.tf: Bump Istio to version 1.29.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/571 (https://phabricator.wikimedia.org/T423726) [19:22:33] (03merge) 10dancy: istio/main.tf: Bump Istio to version 1.29.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/571 (https://phabricator.wikimedia.org/T423726) [19:25:18] (03PS1) 10Thcipriani: castor-save-workspace-cache: exit 0 failing to find directory [integration/config] - 10https://gerrit.wikimedia.org/r/1273935 [19:26:28] ^ Dreamy_Jazz trying this for now, still unsure why this happened. There are three failures for the patchset you pushed for castor-save-cache. BUT none of them think their upstream job were the ones that failed. Needs more investigation. For now, this should fix the failures. [19:27:16] Thanks! [19:27:29] https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/6506075/ and https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/6506067/ and https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/6506066/ were the failures. But not the failures that are reported on your patchset...¯\_(ツ)_/¯ [19:27:51] (03CR) 10Thcipriani: [C:03+2] castor-save-workspace-cache: exit 0 failing to find directory [integration/config] - 10https://gerrit.wikimedia.org/r/1273935 (owner: 10Thcipriani) [19:29:07] in any case, a caching failure should not fail a test run. [19:29:30] (03Merged) 10jenkins-bot: castor-save-workspace-cache: exit 0 failing to find directory [integration/config] - 10https://gerrit.wikimedia.org/r/1273935 (owner: 10Thcipriani) [19:30:14] !log reconfiguring castor-save-workspace-cache with https://gerrit.wikimedia.org/r/1273935 [19:30:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:40:06] (03open) 10dancy: monitoring/values/grafana.yaml.tftpl: Use grafana 12.4.2 image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/572 (https://phabricator.wikimedia.org/T421698) [19:40:07] (03update) 10dancy: monitoring/values/grafana.yaml.tftpl: Use grafana 12.4.2 image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/572 (https://phabricator.wikimedia.org/T421698) [19:41:14] (03merge) 10dancy: monitoring/values/grafana.yaml.tftpl: Use grafana 12.4.2 image [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/572 (https://phabricator.wikimedia.org/T421698) [19:41:23] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Update gitlab-cloud-runners kubernetes - https://phabricator.wikimedia.org/T423726#11834758 (10dancy) p:05Triage→03Low [21:07:40] !log marking integration-agent-1080 offline for experimentation [21:07:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:46:11] 06Project-Admins, 06Quality-and-Test-Engineering-Team: Archive the #Quality-and-Test-Engineering-Team project(?) - https://phabricator.wikimedia.org/T423518#11835091 (10SLong-WMF) a:03SLong-WMF [21:46:32] 06Project-Admins, 06Quality-and-Test-Engineering-Team: Archive the #Quality-and-Test-Engineering-Team project(?) - https://phabricator.wikimedia.org/T423518#11835093 (10SLong-WMF) Assigning this to Armen and myself for collaboration and cleanup [23:30:50] 06Release-Engineering-Team (Seen), 10Release Pipeline (Blubber): Allow new blubber builders to be implemented in yaml - https://phabricator.wikimedia.org/T201875#11835291 (10bd808) I think {96de90b36408f7af977d7d575fba1bcd85880131} implemented this, or at least the start of it.