[00:14:18] (03open) 10dancy: patches.py: Name the patch with a bad "diff --git" line [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1371 [00:15:17] (03update) 10dancy: patches.py: Name the patch with a bad "diff --git" line [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1371 [00:16:32] (03merge) 10dancy: patches.py: Name the patch with a bad "diff --git" line [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1371 [03:00:43] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12452366 (10Eileenmcnaughton) I just merged that rpow fix [03:17:22] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, and 2 others: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12452370 (10pwangai) Thank you @Eileenmcnaughton, with this I believe we have a clear... [05:08:27] 10Gerrit (Gerrit 3.12): Drop shared styles from Zuul plugin once on Gerrit 3.12 - https://phabricator.wikimedia.org/T441756 (10hashar) 03NEW [05:42:35] 10Gerrit, 07Accessibility: Gerrit vote removal usability regression after 3.7 upgrade - https://phabricator.wikimedia.org/T355778#12452563 (10hashar) [06:09:23] (03open) 10marostegui: deployments-calendar.yaml: Remove federico3 [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/275 [06:25:28] 06Project-Admins, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Create user- boards for members of the Security Team (to primarily track security issue responsibilities) - https://phabricator.wikimedia.org/T441335#12452611 (10Aklapper) The project name did not always match the u... [06:35:18] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12452621 (10ABran-WMF) [06:37:45] 10Phabricator, 06Release-Engineering-Team (Radar), 06Product Safety and Integrity, 06Security-Team: Fix ACL team project names - https://phabricator.wikimedia.org/T441760 (10Aklapper) 03NEW [07:11:30] 10Gerrit, 06Gerrit-Privilege-Requests, 06Moderator-Tools-Team, 10XTools: Give access to l10n-bot to labs/xtools - https://phabricator.wikimedia.org/T441761 (10Nikerabbit) 03NEW [07:13:20] 10Continuous-Integration-Config, 10WikiInfo: Please add mediawiki-i18n-check-docker to repository labs/tools/wikiinfo - https://phabricator.wikimedia.org/T441762 (10Nikerabbit) 03NEW [07:46:00] (03update) 10addshore: Work on linti best practices for command docs and structure [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/727 [07:49:12] (03merge) 10addshore: Work on linti best practices for command docs and structure [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/727 [07:56:00] (03update) 10addshore: Draft: Optional development log browsing (T292902) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/722 (https://phabricator.wikimedia.org/T292902) [08:04:15] (03update) 10addshore: Add developer workflow demo recordings [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/726 [08:07:16] (03update) 10addshore: Add developer workflow demo recordings [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/726 [08:15:06] (03CR) 10STran: [C:03+1] Zuul: [mediawiki/extensions/CheckUser] Add WikimediaAntiAbuse dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1356818 (https://phabricator.wikimedia.org/T440871) (owner: 10Mszwarc) [08:20:08] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Add WikimediaAntiAbuse dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1356818 (https://phabricator.wikimedia.org/T440871) (owner: 10Mszwarc) [08:21:24] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Add WikimediaAntiAbuse dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1356818 (https://phabricator.wikimedia.org/T440871) (owner: 10Mszwarc) [08:29:42] (03merge) 10addshore: Add developer workflow demo recordings [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/726 [08:32:48] (03update) 10addshore: Draft: Optional development log browsing (T292902) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/722 (https://phabricator.wikimedia.org/T292902) [08:33:07] (03update) 10addshore: Optional development log browsing (T292902) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/722 (https://phabricator.wikimedia.org/T292902) [08:33:19] (03update) 10addshore: Optional development log browsing [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/722 (https://phabricator.wikimedia.org/T292902) [08:34:14] (03merge) 10addshore: Optional development log browsing [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/722 (https://phabricator.wikimedia.org/T292902) [08:49:17] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Lexicographical data, 06Wikidata-Omega: OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718#12453000 (10ArthurTaylor) We were expecting https://g... [08:52:20] (03CR) 10Hashar: "If I recap we have two lineages of images:" [integration/config] - 10https://gerrit.wikimedia.org/r/1356809 (https://phabricator.wikimedia.org/T420009) (owner: 10Hashar) [08:58:56] (03update) 10addshore: gateway auth for tunnels only, and don't always re tunnel [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/728 [09:01:53] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Lexicographical data, 06Wikidata-Omega (The Board): OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718#12453047 (10ArthurTaylor) [09:02:09] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Lexicographical data, 06Wikidata-Omega (The Board): OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718#12453049 (10ArthurTaylor) a:03ArthurTayl... [09:12:54] hi! we're looking at trying to fix the possible train blocker https://phabricator.wikimedia.org/T441718 [09:13:09] I just merged a patch to master that should fix it: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/1355541 [09:13:34] and would backport that to whatever branch it should be on - I assume the branch cut was already last night [09:14:09] but I don't know what the target branch for the backport should be - wmf/1.48.0-wmf1 doesn't exist yet [09:14:35] should the target be wmf/next? [09:14:47] (03PS1) 10Hashar: Review access change [tools/codesniffer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1357231 [09:15:03] (03PS2) 10Hashar: Deny annotated tags, requiring signed tags [tools/codesniffer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1357231 [09:16:45] (03CR) 10Hashar: [V:03+2 C:03+2] Deny annotated tags, requiring signed tags [tools/codesniffer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1357231 (owner: 10Hashar) [09:19:24] (03CR) 10Hashar: [V:03+2 C:03+2] "There are some tags that did not get signed such as `v49.0.0` or `v51.0.1`" [tools/codesniffer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1357231 (owner: 10Hashar) [09:21:35] codders: you can check the branches from mediawiki/core with git branch -r --list '*/wmf/1*' [09:21:41] origin/wmf/1.47.0-wmf.23 [09:21:41] origin/wmf/1.47.0-wmf.24 [09:21:56] wmf.24 is the one deployed everywhere currently https://versions.toolforge.org/ [09:22:06] and on Tuesday the testwikis should had the wmf.25 deployed [09:22:11] which I guess means that branch did not get cut [09:22:31] okay. so if the patch is now on master (and since 20 mins on master), I don't need to backport it? [09:22:44] or 1.48.0-wmf.1 if we are at the step of bumping [09:22:51] and if that is the case, someone will fix it later tonight I guess [09:22:58] hmm maybe [09:23:12] not sure against which set of commits the new branch will be cut [09:23:30] interesting. that makes two of us :) [09:23:32] and anyway there is no branch to backport to [09:23:38] also that [09:24:26] so if T441718 is a train blocker, you can mention on the train blocker task that it should be verified your fix is indeed included in the new branch [09:24:26] T441718: OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718 [09:24:34] the ticket it marked as a train blocker. I can just write there that I'm not sure if this patch made the cut, and "smarter people" can decide [09:24:42] yeah exactly. [09:24:45] I'll do that - thanks! [09:25:04] Manual! I like it... [09:25:19] Does it make any sense for us to backport to "wmf/next" or is that something internal to releng? [09:26:23] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Lexicographical data, 06Wikidata-Omega (The Board): OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718#12453176 (10ArthurTaylor) I merged the ab... [09:28:48] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774 (10hashar) 03NEW [09:28:56] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453190 (10hashar) p:05Triage→03Unbreak! [09:29:05] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453193 (10hashar) [09:29:12] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222#12453195 (10hashar) The `wmf/1.48.0-wmf.1` branch has not been cut. [09:30:37] codders: 09:29:42 Branching mediawiki/extensions/WikibaseLexeme to wmf/1.48.0-wmf.1 from HEAD: 5cbb83a934e480dc913e22fc3d66e16d7a381c4d [09:30:50] (I retriggered the branch cutting job) [09:31:00] awesome - thank you! [09:33:09] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453255 (10hashar) #collaboration-services can you check the 502 that occurred on Gitlab side at 01:02:12 UTC? ` 01:02:12 stderr: remote: GitL... [09:34:20] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Lexicographical data, 06Wikidata-Omega (The Board): OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718#12453264 (10ArthurTaylor) Per T441774, th... [09:39:07] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453299 (10ABran-WMF) >>! In T441774#12453255, @hashar wrote: > #collaboration-services can you check the 502 that occurred on Gitlab side at... [09:39:31] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Nadia Isanga - https://phabricator.wikimedia.org/T441776 (10Nisangaa) 03NEW [10:10:41] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata Lexicographical data, 06Wikidata-Omega (The Board): OutOfBoundsException: Attempt to get non-existing setting "tmpEnableMobileEditingUIBetaFeature" (Beta Commons down) - https://phabricator.wikimedia.org/T441718#12453431 (10ArthurTaylor) a:05ArthurTayl... [10:32:00] (03update) 10addshore: gateway auth for tunnels only, and don't always re tunnel [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/728 [11:01:48] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12453582 (10ABran-WMF) 05Open→03Resolved p:05Unbreak!→03Medium a:03ABran-WMF Feel free to reopen if needed, that should be fixed :-) [11:03:02] 10GitLab, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12453593 (10ABran-WMF) 05In progress→03Resolved I think this is now fixed? feel free to reopen if needed [11:03:13] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12453598 (10ABran-WMF) [11:23:14] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12453703 (10gmodena) [12:59:53] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Zuul, 10Wikidata: Create a Jenkins job that tests Wikibase against a Mediawiki release version (REL_... branch) - https://phabricator.wikimedia.org/T441825 (10WMDE-leszek) 03NEW [13:01:55] (03PS1) 10C. Scott Ananian: Add a wikimediaDebug option to send an X-Wikimedia-Debug header [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1357275 [13:03:03] (03CR) 10C. Scott Ananian: [C:03+2] Set up user-agent with email id to ensure we aren't throttled [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356957 (owner: 10Subramanya Sastry) [13:03:27] (03CR) 10C. Scott Ananian: [C:03+2] differ.js: Ensure headers are always set [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356958 (owner: 10Subramanya Sastry) [13:03:33] (03Merged) 10jenkins-bot: Set up user-agent with email id to ensure we aren't throttled [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356957 (owner: 10Subramanya Sastry) [13:03:54] (03Merged) 10jenkins-bot: differ.js: Ensure headers are always set [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356958 (owner: 10Subramanya Sastry) [13:04:36] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Zuul, 10Wikidata: Create a Jenkins job that tests Wikibase against a Mediawiki release version (REL_... branch) - https://phabricator.wikimedia.org/T441825#12454414 (10WMDE-leszek) hello @hashar, I'd be intrigued to hear your initial thoughts/reactions... [13:04:53] (03CR) 10Arlolra: [C:03+2] Set up user-agent with email id to ensure we aren't throttled (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356957 (owner: 10Subramanya Sastry) [13:18:35] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12454510 (10hashar) a:05ABran-WMF→03hashar > gitlab was heavily scraped tonight, it is now behind CDN per T425441: gitlab behind CDN so... [13:44:21] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454635 (10trueg) 05Resolved→03In progress Re-opening the ticket. We can now access Gitlab on `eqiad`, `codfw`, how... [13:45:07] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454638 (10ABran-WMF) the fix has been applied to codfw as well but there might be a catch, I'll check [13:45:18] 10GitLab, 06Collaboration-Services, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454639 (10ABran-WMF) p:05Unbreak!→03High [13:49:58] (03PS2) 10Hashar: Archive operations/docker-images/docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/1354921 (https://phabricator.wikimedia.org/T441088) (owner: 10Giuseppe Lavagetto) [13:50:04] (03CR) 10Hashar: Archive operations/docker-images/docker-pkg (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1354921 (https://phabricator.wikimedia.org/T441088) (owner: 10Giuseppe Lavagetto) [13:50:08] 06Release-Engineering-Team, 06Collaboration-Services, 06NFDI-Germany, 10Wikidata, and 3 others: Github actions 403 errors cloning from Gerrit - https://phabricator.wikimedia.org/T441387#12454675 (10ABran-WMF) I asked Claude to test it: ` I could not reproduce it from a residential connection: the exact use... [13:52:05] (03CR) 10Hashar: [C:03+2] Archive operations/docker-images/docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/1354921 (https://phabricator.wikimedia.org/T441088) (owner: 10Giuseppe Lavagetto) [13:52:38] 06Project-Admins, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Create user- boards for members of the Security Team (to primarily track security issue responsibilities) - https://phabricator.wikimedia.org/T441335#12454680 (10sbassett) [13:53:33] (03Merged) 10jenkins-bot: Archive operations/docker-images/docker-pkg [integration/config] - 10https://gerrit.wikimedia.org/r/1354921 (https://phabricator.wikimedia.org/T441088) (owner: 10Giuseppe Lavagetto) [14:02:45] 10Phabricator: Tag (silently) all tasks of T432486 to workboard - https://phabricator.wikimedia.org/T441188#12454758 (10MBinder_WMF) Good questions. :) I think we want all and any subtasks, open or closed, including subsets. I count 62 tasks via a rough copy-paste into a spreadsheet (including the parent task i... [14:04:28] (03CR) 10Daimona Eaytoy: [C:03+1] "> Thus if I understood properly, for both lineages of images:" [integration/config] - 10https://gerrit.wikimedia.org/r/1356809 (https://phabricator.wikimedia.org/T420009) (owner: 10Hashar) [14:06:16] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - https://phabricator.wikimedia.org/T441768#12454769 (10ABran-WMF) The codfw timeouts have the same cause as before, in a form the earlier app... [14:06:38] 10Phabricator, 06Release-Engineering-Team (Radar), 06Product Safety and Integrity, 06Security-Team: Fix ACL team project names - https://phabricator.wikimedia.org/T441760#12454770 (10Catrope) a:03Catrope [14:06:45] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12454781 (10ABran-WMF) [14:07:12] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12454783 (10ABran-WMF) p:05High→03Medium [14:09:13] (03PS1) 10Hashar: opensource: docker-pkg has moved to GitLab [integration/docroot] - 10https://gerrit.wikimedia.org/r/1357294 (https://phabricator.wikimedia.org/T441088) [14:09:33] (03CR) 10Hashar: [C:03+2] opensource: docker-pkg has moved to GitLab [integration/docroot] - 10https://gerrit.wikimedia.org/r/1357294 (https://phabricator.wikimedia.org/T441088) (owner: 10Hashar) [14:09:44] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12454812 (10ABran-WMF) p:05High→03Medium [14:10:10] 10Gerrit, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#12454816 (10sbassett) >>! In T388247#12452109, @Dzahn wrote: > These... [14:10:14] (03Merged) 10jenkins-bot: opensource: docker-pkg has moved to GitLab [integration/docroot] - 10https://gerrit.wikimedia.org/r/1357294 (https://phabricator.wikimedia.org/T441088) (owner: 10Hashar) [14:13:19] 06Release-Engineering-Team, 06Collaboration-Services, 06NFDI-Germany, 10Wikidata, and 3 others: Github actions 403 errors cloning from Gerrit - https://phabricator.wikimedia.org/T441387#12454837 (10ABran-WMF) I'm wondering @hashar if there is any reason for these GH actions to not pull from GH itself? [14:16:19] (03PS1) 10Hashar: opensource: update docker-pkg git repo link [integration/docroot] - 10https://gerrit.wikimedia.org/r/1357301 (https://phabricator.wikimedia.org/T441088) [14:16:32] (03CR) 10Hashar: [C:03+2] opensource: update docker-pkg git repo link [integration/docroot] - 10https://gerrit.wikimedia.org/r/1357301 (https://phabricator.wikimedia.org/T441088) (owner: 10Hashar) [14:17:06] (03Merged) 10jenkins-bot: opensource: update docker-pkg git repo link [integration/docroot] - 10https://gerrit.wikimedia.org/r/1357301 (https://phabricator.wikimedia.org/T441088) (owner: 10Hashar) [14:20:49] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12454904 (10ABran-WMF) [14:22:06] (03Abandoned) 10Hashar: zuul: playbooks for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 (owner: 10Hashar) [14:34:37] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services: Deploy Phab/Phorge 2026-10-20 - https://phabricator.wikimedia.org/T441842 (10brennen) 03NEW [14:36:21] (03open) 10aklapper: Merge remote-tracking branch 'phorge/master' into work/2026-10-20-merge-phorge [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/131 (https://phabricator.wikimedia.org/T441842) [14:36:34] (03open) 10aklapper: Merge remote-tracking branch 'arcanist/master' into work/2026-10-20-merge-phorge [repos/phabricator/arcanist] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/arcanist/-/merge_requests/10 (https://phabricator.wikimedia.org/T441842) [14:36:55] (03update) 10aklapper: Merge remote-tracking branch 'phorge/master' into work/2026-10-20-merge-phorge [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/131 (https://phabricator.wikimedia.org/T441842) [14:44:38] (03update) 10aklapper: Merge remote-tracking branch 'phorge/master' into work/2026-10-20-merge-phorge [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/131 (https://phabricator.wikimedia.org/T441842) [14:50:58] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Deploy Phab/Phorge 2026-10-20 - https://phabricator.wikimedia.org/T441842#12455090 (10Aklapper) Rebasing and merging upstream, list of merge conflicts: Phorge: * both modified: resources/celerity/m... [14:53:23] (03update) 10brennen: Merge remote-tracking branch 'phorge/master' into work/2026-10-20-merge-phorge [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/131 (https://phabricator.wikimedia.org/T441842) (owner: 10aklapper) [14:53:26] (03update) 10brennen: Merge remote-tracking branch 'phorge/master' into work/2026-10-20-merge-phorge [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/131 (https://phabricator.wikimedia.org/T441842) (owner: 10aklapper) [14:53:31] (03update) 10brennen: Merge remote-tracking branch 'phorge/master' into work/2026-10-20-merge-phorge [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/131 (https://phabricator.wikimedia.org/T441842) (owner: 10aklapper) [14:53:44] (03merge) 10brennen: Merge remote-tracking branch 'phorge/master' into work/2026-10-20-merge-phorge [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/131 (https://phabricator.wikimedia.org/T441842) (owner: 10aklapper) [14:54:07] (03merge) 10brennen: Merge remote-tracking branch 'arcanist/master' into work/2026-10-20-merge-phorge [repos/phabricator/arcanist] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/arcanist/-/merge_requests/10 (https://phabricator.wikimedia.org/T441842) (owner: 10aklapper) [14:54:23] (03open) 10dancy: Release 4.302.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1372 [14:56:35] 10Deployments, 10GitLab, 06Release-Engineering-Team, 06Collaboration-Services: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774#12455123 (10dancy) Thanks for fixing things up @hashar! [14:57:56] (03merge) 10dancy: Release 4.302.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1372 [14:59:33] (03open) 10brennen: update submodules for 2026-10-20 deploy (includes upstream merge) [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/122 (https://phabricator.wikimedia.org/T441842) [15:00:38] (03merge) 10brennen: update submodules for 2026-10-20 deploy (includes upstream merge) [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/122 (https://phabricator.wikimedia.org/T441842) [15:00:53] (03CR) 10C. Scott Ananian: Add a new config for php8.3 vs php8.5 testing (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356959 (https://phabricator.wikimedia.org/T440321) (owner: 10Subramanya Sastry) [15:01:14] (03Abandoned) 10C. Scott Ananian: Add a wikimediaDebug option to send an X-Wikimedia-Debug header [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1357275 (owner: 10C. Scott Ananian) [15:09:06] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:09:14] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T441851 (10wmcs-alerts) 03NEW [15:20:08] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455288 (10trueg) `wdqs-proxy` [[ https://gitlab.wikimedia.org/repos/wikidata-pla... [15:26:10] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Deploy Phab/Phorge 2026-10-20 - https://phabricator.wikimedia.org/T441842#12455308 (10Aklapper) FYI We did not deploy today as we ran into some scap / keyholder issues on the devtools test machine. O... [15:26:55] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455315 (10trueg) @ABran-WMF The only thing I do not understand is the need for t... [15:30:08] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: Deploy Phab/Phorge 2026-10-20 - https://phabricator.wikimedia.org/T441842#12455327 (10brennen) [15:30:38] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455330 (10trueg) CR https://gerrit.wikimedia.org/r/c/operations/deployment-chart... [15:31:48] 10GitLab, 06Collaboration-Services, 13Patch-For-Review, 06Wikidata Platform Team (Sprint 10 (2026/10/07)): [BUG] wdqs deployment times out (gitlab is not reachable) - only on codfw - https://phabricator.wikimedia.org/T441768#12455335 (10trueg) 05In progress→03Resolved [15:34:51] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Test Platform, 07Epic: Implement CI dashboards (https://releng-data.wmcloud.org/-/dashboards) for the new Zuul setup - https://phabricator.wikimedia.org/T441692#12455346 (10AMarkossyan-WMF) The new zuul is running but there's no data that we can get b... [15:52:07] 06Gerrit-Privilege-Requests, 10MediaWiki-extensions-DocBookExport: Want +2 right for myself (Yaron_Koren) for DocBookExport extension - https://phabricator.wikimedia.org/T441856 (10Yaron_Koren) 03NEW [16:03:06] 06Release-Engineering-Team, 06Collaboration-Services, 06NFDI-Germany, 10Wikidata, and 3 others: Github actions 403 errors cloning from Gerrit - https://phabricator.wikimedia.org/T441387#12455476 (10hashar) > I'm wondering @hashar if there is any reason for these GH actions to not pull from GH itself? Beca... [16:36:59] (03PS4) 10Subramanya Sastry: Add a new config for php8.3 vs php8.5 testing [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356959 (https://phabricator.wikimedia.org/T440321) [16:36:59] (03PS1) 10Subramanya Sastry: Take 2: differ.js: Ensure headers are always set [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1357332 [16:37:49] (03CR) 10Subramanya Sastry: Add a new config for php8.3 vs php8.5 testing (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356959 (https://phabricator.wikimedia.org/T440321) (owner: 10Subramanya Sastry) [16:39:14] (03CR) 10Subramanya Sastry: [C:03+2] Take 2: differ.js: Ensure headers are always set [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1357332 (owner: 10Subramanya Sastry) [16:40:08] (03CR) 10Subramanya Sastry: [C:03+2] Add a new config for php8.3 vs php8.5 testing [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356959 (https://phabricator.wikimedia.org/T440321) (owner: 10Subramanya Sastry) [16:40:11] (03Merged) 10jenkins-bot: Take 2: differ.js: Ensure headers are always set [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1357332 (owner: 10Subramanya Sastry) [16:41:03] (03Merged) 10jenkins-bot: Add a new config for php8.3 vs php8.5 testing [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1356959 (https://phabricator.wikimedia.org/T440321) (owner: 10Subramanya Sastry) [16:41:53] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 06Product Safety and Integrity, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216#12455630 (10sbassett) [16:42:31] (03update) 10dancy: patches: Comment on the task of an already-applied security patch [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1370 (https://phabricator.wikimedia.org/T439619) [17:08:14] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12455834 (10bd808) I haven't had time to try and make my logging move verbose yet, but {T441864} seems very likely directly related to the cutover announ... [17:18:36] hey folks, wondering if we have written down anywhere the preferred choices for key algorithm/length/expiry for https://www.mediawiki.org/keys/keys.html (for signing release artifacts), and if we don't, what the recommendations are/should be. asking y'all first as connected with the release process, feel free to send me elsewhere if needed [17:32:43] 06Release-Engineering-Team: Request to change Phabricator name - https://phabricator.wikimedia.org/T441868 (10Giorgione1998) 03NEW [17:33:42] 10Phabricator: Tag (silently) all tasks of T432486 to workboard - https://phabricator.wikimedia.org/T441188#12455956 (10Aklapper) The difference are whatever tasks got added within the last four days. :P [17:44:49] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Nadia Isanga - https://phabricator.wikimedia.org/T441776#12456031 (10brennen) 05Open→03Resolved a:03brennen [17:45:02] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for Nadia Isanga - https://phabricator.wikimedia.org/T441776#12456040 (10brennen) Approved. Happy hacking! [17:46:05] 10Phabricator: Request to change Phabricator name - https://phabricator.wikimedia.org/T441868#12456043 (10Aklapper) [17:46:26] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Request to change Phabricator name - https://phabricator.wikimedia.org/T441868#12456044 (10Aklapper) 05Open→03Resolved a:03Aklapper Done [18:14:38] 10Gerrit, 06Collaboration-Services, 13Patch-For-Review, 07Upstream: Gerrit deployment initialize the site generating ssh key pairs - https://phabricator.wikimedia.org/T441113#12456229 (10Dzahn) 05Open→03Resolved config change deployed. With this I would summarize: - upstream patch is enough - upg... [18:15:21] 10Gerrit, 06Collaboration-Services, 13Patch-For-Review, 07Upstream: Gerrit deployment initialize the site generating ssh key pairs - https://phabricator.wikimedia.org/T441113#12456232 (10Dzahn) well, the only exception being that if there are any other hard restarts _before_ the upgrade to 3.12 we stil... [18:58:31] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222#12456487 (10dancy) [19:14:54] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/VisualEditor] Add AbuseFilter and SpamBlacklist Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1357377 [19:16:53] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/VisualEditor] Add AbuseFilter and SpamBlacklist Phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1357377 (owner: 10Fomafix) [19:19:00] (03PS2) 10Fomafix: Zuul: [mediawiki/extensions/VisualEditor] Add Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1357377 [20:08:24] !log Remove cherry-pick for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341351 to fix merge conflict - T441851 [20:08:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:08:29] T441851: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T441851 [20:14:06] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [21:05:16] !log Beta Cluster: allow 873/tcp to swift-fe security group to fix fetch-rings-deployment-prep.service on Puppetserver - T436476 [21:05:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:05:20] T436476: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476 [21:48:49] !log Beta Cluster: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1357408 - T436476 [21:48:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:48:53] T436476: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476 [22:59:30] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/125 (T438720) [22:59:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:59:34] T438720: Code server image should not run curl'd shellcode - https://phabricator.wikimedia.org/T438720