[03:15:50] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge upstream / Arcanist upstream (after 2025-09-08 done in T404134) - https://phabricator.wikimedia.org/T404375#11223117 (10Nobchanreth) [04:31:58] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge upstream / Arcanist upstream (after 2025-09-08 done in T404134) - https://phabricator.wikimedia.org/T404375#11223136 (10JJMC89) [08:16:28] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11223446 (10Jelto) Sync from object storage to a local folder works with the new `ceph::client::sync_loca... [08:28:30] 10GitLab (Project Migration), 06Community-Tech, 10SVG Translate Tool, 13Patch-For-Review: Migrate SVG Translate from GitHub to GitLab - https://phabricator.wikimedia.org/T402505#11223503 (10Samwilson) 05In progress→03Open [08:45:34] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11223644 (10awight) @dancy Thanks for doing this investigation! I have a hard time following exac... [08:58:08] (03open) 10brouberol: Add repos/data-engineering/kafka to the trusted repos [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/133 [09:12:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:12:36] 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-maps-master02 in project deployment-prep - https://phabricator.wikimedia.org/T405888 (10wmcs-alerts) 03NEW [09:40:16] I find myself needing to inspect a k8s image under "restricted/". Is there a debug/noncritical host in the production cluster where I can do that? [09:50:55] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11223925 (10awight) 05Open→03Resolved a:03awight I see that the number at the end is not... [10:08:36] I... no longer have this need, only a lingering curiosity. nvm! [10:46:21] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Ruvchoudhary - https://phabricator.wikimedia.org/T404484#11224101 (10Aklapper) 05Open→03Declined Unfortunately closing this Phabricator task as no further information has been provided. @Ruvchoudhary: Aft... [10:48:06] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for 3mindedscholar - https://phabricator.wikimedia.org/T405832#11224108 (10Aklapper) 05Open→03Resolved a:03Aklapper Done! [12:15:52] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11224409 (10Jelto) [12:23:55] (03update) 10brouberol: Add repos/data-engineering/kafka to the trusted repos [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/133 [12:29:08] (03update) 10brouberol: Add repos/data-engineering/kafka to the trusted repos [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/133 [13:35:04] (03PS1) 10Hashar: dockerfiles: stop running ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1192134 [13:35:05] (03PS1) 10Hashar: jjb: update java jobs to stop running ci-src-setup twice [integration/config] - 10https://gerrit.wikimedia.org/r/1192135 [13:35:23] (03CR) 10Hashar: [C:03+2] dockerfiles: stop running ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1192134 (owner: 10Hashar) [13:37:38] (03Merged) 10jenkins-bot: dockerfiles: stop running ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1192134 (owner: 10Hashar) [13:38:17] 06Project-Admins: Tag for community collab tasks - https://phabricator.wikimedia.org/T405909 (10Nikerabbit) 03NEW [13:43:26] !jouncebot now [13:43:57] (oops, it's in #wikimedia-operations) [13:44:59] (03CR) 10Hashar: "recheck after I66b8140e9d07af2f6616920404588cad7307a1b5" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/1189952 (https://phabricator.wikimedia.org/T201491) (owner: 10Raveen Shalom Rodney) [13:46:29] !log Updated Java jobs to images that no more run ci-src-setup | https://gerrit.wikimedia.org/r/c/integration/config/+/1192134/ | https://gerrit.wikimedia.org/r/c/integration/config/+/1192135/ [13:46:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:47:24] (03CR) 10Hashar: [C:03+2] "Verified on https://gerrit.wikimedia.org/r/c/integration/gearman-java/+/1189952. I have updated the jobs and did a `!log` for it." [integration/config] - 10https://gerrit.wikimedia.org/r/1192135 (owner: 10Hashar) [13:50:00] (03Merged) 10jenkins-bot: jjb: update java jobs to stop running ci-src-setup twice [integration/config] - 10https://gerrit.wikimedia.org/r/1192135 (owner: 10Hashar) [13:56:32] (03CR) 10Hashar: [C:03+2] "@raveenshalomrodney@gmail.com thank you for the fix! CI is a bit broken because that project is not as actively maintained as it used to b" [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/1189952 (https://phabricator.wikimedia.org/T201491) (owner: 10Raveen Shalom Rodney) [13:59:48] (03Merged) 10jenkins-bot: fix: correct variable name typo in GearmanJobStatusImpl [integration/gearman-java] - 10https://gerrit.wikimedia.org/r/1189952 (https://phabricator.wikimedia.org/T201491) (owner: 10Raveen Shalom Rodney) [14:48:39] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11224950 (10Jelto) [14:49:10] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#11224955 (10Jelto) [15:00:03] (03merge) 10dancy: Add repos/data-engineering/kafka to the trusted repos [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/133 (owner: 10brouberol) [15:05:26] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11225032 (10dancy) >>! In T405738#11223644, @awight wrote: > @dancy Thanks for doing this inve... [15:30:11] (03update) 10dancy: kubernetes.py: Fix inspect_images() with empty list [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1005 [15:33:21] (03merge) 10dancy: kubernetes.py: Fix inspect_images() with empty list [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1005 [15:35:08] (03update) 10dancy: Release 4.211.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1006 [15:35:10] (03open) 10dancy: Release 4.211.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1006 [15:37:26] (03merge) 10dancy: Release 4.211.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1006 [15:46:16] 06Project-Admins: Create project tag for Wanda - https://phabricator.wikimedia.org/T405923 (10Techwizzie) 03NEW [15:48:34] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Test new zuul test VMs - https://phabricator.wikimedia.org/T405120#11225238 (10LSobanski) p:05Triage→03High [15:48:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Test new zuul test VMs - https://phabricator.wikimedia.org/T405120#11225240 (10LSobanski) p:05High→03Triage [15:48:41] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul scheduler on zuul1001 - https://phabricator.wikimedia.org/T405118#11225241 (10LSobanski) p:05Triage→03High [15:49:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul scheduler on zuul1001 - https://phabricator.wikimedia.org/T405118#11225243 (10LSobanski) a:03Dzahn [15:54:20] 06Project-Admins: Create project tag for Wikisource Reader App - https://phabricator.wikimedia.org/T405599#11225305 (10Dzahn) Great, sounds good. Thank you [15:56:40] !log Upgrading scap to 4.211.0 in beta cluster [15:56:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:15] Project beta-scap-sync-world build #225944: 04FAILURE in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/225944/ [15:57:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001 - https://phabricator.wikimedia.org/T405119#11225321 (10Dzahn) [15:58:31] 06Release-Engineering-Team, 10Scap, 10Cite (Sub-referencing), 05MW-1.45-notes (1.45.0-wmf.20; 2025-09-23), and 2 others: Debug scap partial deployment, 25 Sept 2025 - https://phabricator.wikimedia.org/T405738#11225329 (10dancy) The fix for the incorrect commit info in https://test.wikipedia.org/wiki/Sp... [16:02:00] Yippee, build fixed! [16:02:00] Project beta-scap-sync-world build #225945: 09FIXED in 2 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/225945/ [16:05:27] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001 - https://phabricator.wikimedia.org/T405119#11225374 (10LSobanski) p:05Triage→03High a:03Dzahn [16:06:21] (03open) 10volker-e: releases: Bump Codex to 1.2.34 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/93 (https://phabricator.wikimedia.org/T405501) [16:06:34] (03update) 10volker-e: releases: Bump Codex to 2.3.2 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/93 (https://phabricator.wikimedia.org/T405501) [16:33:49] (03approved) 10lmora: releases: Bump Codex to 2.3.2 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/93 (https://phabricator.wikimedia.org/T405501) (owner: 10volker-e) [16:34:00] (03merge) 10lmora: releases: Bump Codex to 2.3.2 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/93 (https://phabricator.wikimedia.org/T405501) (owner: 10volker-e) [16:48:24] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Deploy zuul executor on executor VM - https://phabricator.wikimedia.org/T403847#11225724 (10Dzahn) 05In progress→03Resolved I am claiming this as resolved.. an executor i... [16:49:08] (03update) 10dancy: kubernetes.py: Handle multiple clusters better [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/999 (https://phabricator.wikimedia.org/T405110) [16:51:22] (03merge) 10dancy: kubernetes.py: Handle multiple clusters better [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/999 (https://phabricator.wikimedia.org/T405110) [16:53:43] (03open) 10dancy: Release 4.212.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1007 [16:55:26] (03merge) 10dancy: Release 4.212.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1007 [17:38:23] 10Scap, 13Patch-For-Review: Allow the same namespace name to be used in different clusters - https://phabricator.wikimedia.org/T405110#11225975 (10dancy) 05Open→03Resolved Changes have been deployed and tested. [17:47:57] (03open) 10dancy: scap clean-images: sudo to the docker_user if needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1008 (https://phabricator.wikimedia.org/T387927) [17:47:58] (03update) 10dancy: scap clean-images: sudo to the docker_user if needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1008 (https://phabricator.wikimedia.org/T387927) [17:51:08] (03update) 10dancy: scap clean-images: sudo to the docker_user if needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1008 (https://phabricator.wikimedia.org/T387927) [17:52:21] (03merge) 10dancy: scap clean-images: sudo to the docker_user if needed [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1008 (https://phabricator.wikimedia.org/T387927) [17:54:03] (03open) 10dancy: Release 4.213.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1009 [17:56:04] (03merge) 10dancy: Release 4.213.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1009 [18:59:29] 10Phabricator (Upstream), 06Release-Engineering-Team (Priority Backlog 📥): PHP Warning: Missing boundary in multipart/form-data POST data in Unknown on line 0 - https://phabricator.wikimedia.org/T396869#11226268 (10Aklapper) 05Open→03Declined I've come to the conclusion that this is ignorable. [19:00:29] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-puppetserver-1 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:00:47] 06Project-Admins: Create project tag for Wanda - https://phabricator.wikimedia.org/T405923#11226282 (10Aklapper) @Techwizzie: Hi, do you plan to disable Issues at https://github.com/WikiWorks/Wanda/issues and move issue tracking completely to Phabricator? [19:10:01] 06Project-Admins: Tag for community collab tasks - https://phabricator.wikimedia.org/T405909#11226316 (10Aklapper) Closest coming to my mind is are the two "Community Feedback" subprojects in Reader/Growth... Could you please provide a project description and tag type? See https://www.mediawiki.org/wiki/Phabric... [19:11:29] 06Project-Admins, 10Product-Platform Operations, 10Projects-Cleanup: Status of Platform-Product Steering Committee - https://phabricator.wikimedia.org/T404313#11226328 (10Aklapper) @Naike: Would you know the status of this, or have an idea who would know? Thanks! [19:57:52] dancy: is puppet still broken on that beta cluster vps? [20:03:39] cdanis: I will check [20:04:29] Looks like puppet ran successfully 29 minutes ago. [20:05:06] And I just ran run-puppet-agent successfully now. [20:07:33] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul scheduler on zuul1001 - https://phabricator.wikimedia.org/T405118#11226759 (10Dzahn) First attempt to start this new systemd unit. Currently fails with: ` Sep 29 20:06:08 zu... [20:07:35] neat [20:07:59] well I finished writing a patch which I still think is an improvement, so 😅 [20:09:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work, 13Patch-For-Review: Set up zuul scheduler on zuul1001 - https://phabricator.wikimedia.org/T405118#11226761 (10Dzahn) is this what is missing? ` python3-wmfdb - Libraries for interacting with WMF's mariadb d... [20:10:28] Thank you for doing that [23:10:41] is there any way to tell whether my fix for T405893 made MW CI faster on average? [23:10:43] T405893: Slow test UploadFromUrlTest::testSyncDownload due to PHPUnit enumerator - https://phabricator.wikimedia.org/T405893 [23:23:38] TimStarling: I don't think we keep metrics for phpunit, just for jobs overall, and bigger machine-level/project-level on, e.g., https://tools-static.wmflabs.org/jenkins-build-stats/2025/09/2025-09-28/1-day/ it might be possible to get this number looking at log output on the jenkins server. [23:29:58] thanks, that's helpful [23:33:39] I think if the build time fell by 5 minutes then that would probably be visible in the overall job execution times. But I need to wait for the next day (maybe 2 days) of CSV file to really tell [23:37:53] reading that task, looks like a nice speed up. Hopefully it'll be visible in per-project data.