[06:46:12] mutante: Good morning! My team is enabling a new wikitext syntax element (sub-referencing) to dewiki and it occurs to me that we might want a dedicated window, eg at 08h UTC. Does this seem reasonable? [06:59:26] mutante: placeholder: https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250922T0800 [07:21:23] he's not on the releng team ;) [07:21:28] and almost certainly asleep [07:36:36] Reedy: hehe sorry this was based on the "clinic duty" topic message in -operations [08:20:17] (03CR) 10Tchanders: "Looks good to me" [integration/config] - 10https://gerrit.wikimedia.org/r/1187443 (https://phabricator.wikimedia.org/T264359) (owner: 10Tchanders) [09:47:48] awight: Clinic duty is for SRE things, not RelEng / production release concerns. [09:52:50] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 10Projects-Cleanup, 10MediaWiki-Platform-Team (Radar): Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11200872 (10Jdforrester-WMF) [10:10:01] James_F: +1 I wonder how to ping the most awake releng person in this channel—in the past I've just used the same person all the time, but maybe there's a shift calendar I should watch? [10:10:09] !log sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node20-docs-publish/ # T405166 [10:10:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:10:14] T405166: mwext-node20-docs-publish failing post-merge for CodeMirror - https://phabricator.wikimedia.org/T405166 [10:10:40] awight: RelEng aren't an on-call team. Ask in here, don't ping a human, wait for a response. Or file a Phabricator task like everyone else. :-) [10:12:19] James_F: for a concrete use case, today we realized that our change was too major for the regular backport window so we looked at the docs here https://wikitech.wikimedia.org/wiki/Deployments/Inclusion_criteria [10:12:47] We decided to create a special, one-off deployment window but the instructions are not quite clear IMO about how to communicate this. [10:13:26] I think they're pretty clear. You need to get approval from RelEng. It's not an emergency deployment in this circumstance, so pinging is not appropriate? [10:14:03] Where are you seeing "need to get approval from RelEng"? [10:15:53] "If in doubt, talk to Release Engineering.". You're in doubt. [10:16:05] Anyway, there isn't a problem from my perspective. m-utante will have a mysterious message upon waking but the rest of the chat history should dispel any doubts. [10:16:18] Sorry, the "you" was not "one needs to" but you in particular seem to need to. [10:16:51] I think it's fine to proceed without, assuming all the involved stakeholders are aware this is happening and OK (so MW group, SRE, Editing, etc.?). [10:18:05] I'm... okay, we unilaterally decided on a special window and nothing was harmed. I might chat with RelEng about how to clarify the docs in the future or I might just sneak into deployment tunnels with a small wrench ;-) [10:18:45] I would discourage going rogue and finding out what it takes to lose deployment privileges. :-) [10:19:59] (03CR) 10Dreamy Jazz: "Looks good to me too" [integration/config] - 10https://gerrit.wikimedia.org/r/1187443 (https://phabricator.wikimedia.org/T264359) (owner: 10Tchanders) [10:20:18] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-extensions-CodeMirror: mwext-node20-docs-publish failing post-merge for CodeMirror - https://phabricator.wikimedia.org/T405166#11200920 (10Jdforrester-WMF) Should now be fixed. > I'd say for CodeMirror, we need to bust the cache maybe... [10:22:49] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Remove dependency on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1187443 (https://phabricator.wikimedia.org/T264359) (owner: 10Tchanders) [10:26:03] James_F: oddly misplaced threats aside... my original point is that the docs do not currently specify what is rogue behavior [10:26:48] I'm confused. Are you talking about your threat to "sneak into deployment tunnels"? [10:28:15] Multiple threats were made, let's leave this all aside now. [10:28:32] But also, I don't think trying to write a prescriptivist view of what mis- or ab-use of deployment rights is helpful. The English Wikipedia guideline of WP:BEANS applies. [10:30:58] The only question I am interested in at the moment is, https://wikitech.wikimedia.org/wiki/Deployments/Inclusion_criteria#How_to_add_a_special_deployment_window in this section perhaps there should be a step for "communicate your new special window to X", that is really all. [10:31:59] Oh, indeed. As I said above, you should file a Phabricator task. [10:34:07] (03PS5) 10Jforrester: Zuul: [mediawiki/extensions/CheckUser] Remove dependency on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1187443 (https://phabricator.wikimedia.org/T264359) (owner: 10Tchanders) [10:34:10] (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1187443 (https://phabricator.wikimedia.org/T264359) (owner: 10Tchanders) [10:34:13] <3 this might happen [10:34:39] Excellent. I think you're right that more re-assurance is warranted in the docs. [10:36:50] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Remove dependency on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1187443 (https://phabricator.wikimedia.org/T264359) (owner: 10Tchanders) [10:37:55] (03CR) 10Isabelle Hurbain-Palatin: [C:03+2] Update diffs.sh script: tweaks + dump topfails links for wikis [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1189966 (owner: 10Subramanya Sastry) [10:38:42] (03Merged) 10jenkins-bot: Update diffs.sh script: tweaks + dump topfails links for wikis [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1189966 (owner: 10Subramanya Sastry) [10:40:08] !log Zuul: [mediawiki/extensions/CheckUser] Remove dependency on GuidedTour, for T264359 [10:40:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:40:10] T264359: Remove GuidedTour from Special:Investigate - https://phabricator.wikimedia.org/T264359 [10:42:17] (03PS3) 10Jforrester: jjb, Zuul: [labs/tools/heritage] Switch python 3.7 for 3.9 in publish job [integration/config] - 10https://gerrit.wikimedia.org/r/1189957 (owner: 10Lokal Profil) [10:42:59] (03PS4) 10Jforrester: jjb, Zuul: [labs/tools/heritage] Switch python 3.7 for 3.9 in publish job [integration/config] - 10https://gerrit.wikimedia.org/r/1189957 (owner: 10Lokal Profil) [10:43:37] (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [labs/tools/heritage] Switch python 3.7 for 3.9 in publish job [integration/config] - 10https://gerrit.wikimedia.org/r/1189957 (owner: 10Lokal Profil) [10:43:43] (03CR) 10Jforrester: [C:03+2] "jjb changes deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1189957 (owner: 10Lokal Profil) [10:45:16] (03Merged) 10jenkins-bot: jjb, Zuul: [labs/tools/heritage] Switch python 3.7 for 3.9 in publish job [integration/config] - 10https://gerrit.wikimedia.org/r/1189957 (owner: 10Lokal Profil) [10:48:52] !log Zuul: [labs/tools/heritage] Switch python 3.7 for 3.9 in publish job [10:48:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:14:06] (03PS2) 10Bartosz Wójtowicz: inference-services: Remove the outlink-transformer CI job. [integration/config] - 10https://gerrit.wikimedia.org/r/1187752 (https://phabricator.wikimedia.org/T404294) [11:14:28] (03CR) 10Bartosz Wójtowicz: "We've removed the transformer part from both staging and production clusters today, everything is working smoothly after the deployments." [integration/config] - 10https://gerrit.wikimedia.org/r/1187752 (https://phabricator.wikimedia.org/T404294) (owner: 10Bartosz Wójtowicz) [11:16:45] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Change Maniphest AdvancedSearch default for "Group by" to "None" and for "Order by" to "Updated" - https://phabricator.wikimedia.org/T404843#11201034 (10Aklapper) p:05Triage→03Low [11:16:56] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Change Maniphest AdvancedSearch default for "Group by" to "None" and for "Order by" to "Updated" - https://phabricator.wikimedia.org/T404843#11201037 (10Aklapper) I agree that Priority as a default in our installation makes less sense than in that... [11:28:25] (03PS4) 10Jforrester: Zuul: Adjust MW change-queue to explicitly apply to extensions/skins [integration/config] - 10https://gerrit.wikimedia.org/r/1188391 [11:28:32] (03CR) 10Jforrester: [C:03+2] "Acknowledged" [integration/config] - 10https://gerrit.wikimedia.org/r/1188391 (owner: 10Jforrester) [11:30:25] (03Merged) 10jenkins-bot: Zuul: Adjust MW change-queue to explicitly apply to extensions/skins [integration/config] - 10https://gerrit.wikimedia.org/r/1188391 (owner: 10Jforrester) [11:47:35] (03CR) 10AikoChou: [C:03+1] inference-services: remove nsfw pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/1189843 (https://phabricator.wikimedia.org/T405083) (owner: 10Ilias Sarantopoulos) [12:15:30] (03PS1) 10Kosta Harlan: zuul: GrowthExperiments depends on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) [12:16:20] 06Release-Engineering-Team, 06Growth-Team, 10GrowthExperiments, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: GrowthExperiments: AddImage.cy.ts and AddALink.cy.ts fail in CI when GuidedTour is not installed - https://phabricator.wikimedia.org/T405220#11201200 (10kostajh) [12:17:11] (03PS2) 10Kosta Harlan: zuul: GrowthExperiments depends on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) [12:18:33] (03CR) 10CI reject: [V:04-1] zuul: GrowthExperiments depends on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) (owner: 10Kosta Harlan) [12:18:38] (03CR) 10Dreamy Jazz: [C:03+1] zuul: GrowthExperiments depends on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) (owner: 10Kosta Harlan) [12:19:01] (03PS3) 10Kosta Harlan: zuul: GrowthExperiments depends on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) [12:20:00] 06Release-Engineering-Team, 06Growth-Team, 10GrowthExperiments, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: GrowthExperiments: AddImage.cy.ts and AddALink.cy.ts fail in CI when GuidedTour is not installed - https://phabricator.wikimedia.org/T405220#11201205 (10Dreamy_Jazz) a:03kost... [12:37:55] (03CR) 10Jforrester: [C:03+2] zuul: GrowthExperiments depends on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) (owner: 10Kosta Harlan) [12:38:14] (03PS4) 10Jforrester: Zuul: [mediawiki/extensions/GrowthExperiments] Add dependency on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) (owner: 10Kosta Harlan) [12:38:18] (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) (owner: 10Kosta Harlan) [12:40:04] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/GrowthExperiments] Add dependency on GuidedTour [integration/config] - 10https://gerrit.wikimedia.org/r/1190263 (https://phabricator.wikimedia.org/T405220) (owner: 10Kosta Harlan) [12:40:26] !log Zuul: [mediawiki/extensions/GrowthExperiments] Add dependency on GuidedTour, for T405220 [12:40:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:40:29] T405220: GrowthExperiments: AddImage.cy.ts and AddALink.cy.ts fail in CI when GuidedTour is not installed - https://phabricator.wikimedia.org/T405220 [12:40:40] kostajh, Dreamy_Jazz: Deployed now, hopefully this fixes. [12:42:31] Thanks [12:46:20] 06Release-Engineering-Team, 06Growth-Team, 10GrowthExperiments, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: GrowthExperiments: AddImage.cy.ts and AddALink.cy.ts fail in CI when GuidedTour is not installed - https://phabricator.wikimedia.org/T405220#11201275 (10Jdforrester-WMF) ... [13:01:19] (03PS3) 10Jforrester: build: Upgrade eslint-config-wikimedia to 0.31.0 and make pass [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1176753 [13:01:54] (03CR) 10CI reject: [V:04-1] build: Upgrade eslint-config-wikimedia to 0.31.0 and make pass [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1176753 (owner: 10Jforrester) [13:02:36] James_F: thanks! [13:02:49] (03CR) 10Jforrester: "I've re-generated test/fixtures/basic-records/one/record.json but this seems like quite a set of changes — is this correct?" [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1176753 (owner: 10Jforrester) [13:18:14] 10Continuous-Integration-Config, 10MediaWiki-libs-IPUtils, 10Projects-Cleanup, 10MediaWiki-Platform-Team (Radar): Archive wikimedia/ip-set library - https://phabricator.wikimedia.org/T401155#11201415 (10Krinkle) [14:04:58] 10Phabricator (Upstream), 07Regression: No notification is sent to a task when a new subtask of it is created & that new subtask's status is "Open" - https://phabricator.wikimedia.org/T405084#11201594 (10A_smart_kitten) [14:07:00] !log ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node20-rundoc # fix failure seen in mwext-node20-rundoc 18248 + 18249 [14:07:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:07:05] 10Phabricator (Upstream), 07Regression, 07Upstream: No notification is sent to a task when a new subtask of it is created & that new subtask's status is "Open" - https://phabricator.wikimedia.org/T405084#11201601 (10A_smart_kitten) [15:04:39] 10Phabricator: Request to re-enable Atitkov acccount - https://phabricator.wikimedia.org/T405237 (10Jdrewniak) 03NEW [15:07:43] Lucas_WMDE: Oh no, again? I did that just a few hours ago. [15:08:08] James_F: different job, yours was mwext-node20-docs-publish [15:08:14] (unless you also did another rm -rf that didn’t end up in SAL) [15:08:15] Oh, duh. Yeah. [15:08:20] Nope. [15:08:22] Ignore me. :-) [15:08:26] but yeah, not fun :S [15:39:55] 10Phabricator: Request to re-enable Atitkov acccount - https://phabricator.wikimedia.org/T405237#11202106 (10Reedy) 05Open→03Resolved a:03Reedy Done! [15:46:07] 10Phabricator: Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245 (10Sdkb) 03NEW [16:14:36] (03CR) 10Krinkle: "recheck" [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1022025 (owner: 10Majavah) [16:16:57] (03CR) 10Jforrester: "Hmm, this passes locally on fresh-node20. Interesting." [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1176753 (owner: 10Jforrester) [16:24:28] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [16:28:04] (03PS1) 10Matthias Mullie: Add WikimediaMessages as dependency for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1190306 [16:32:44] (03CR) 10Krinkle: "It seems something (?) has started to cause fractions to be lost, however not always. Most test cases still correctly transfer, preserve, " [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1176753 (owner: 10Jforrester) [16:35:01] (03CR) 10Jforrester: "> That leaves us with... the npm-shrinkwrap change. Maybe npm updated an indirect non-dev dependencies from Puppeteer at the same time? Th" [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1176753 (owner: 10Jforrester) [16:36:30] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/ReaderExperiments] Add WikimediaMessages dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1190306 (owner: 10Matthias Mullie) [16:39:03] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ReaderExperiments] Add WikimediaMessages dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1190306 (owner: 10Matthias Mullie) [16:40:34] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ReaderExperiments] Add WikimediaMessages dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1190306 (owner: 10Matthias Mullie) [16:41:04] !log Zuul: [mediawiki/extensions/ReaderExperiments] Add WikimediaMessages dependency [16:41:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:18:40] (03CR) 10Krinkle: "I dread the possibility of this being some low-level Rosetta Intel-to-ARM magic going wrong. Let's hope it's something else but yeah let m" [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1176753 (owner: 10Jforrester) [17:44:00] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10MediaWiki-extensions-CodeMirror: mwext-node20-docs-publish failing post-merge for CodeMirror - https://phabricator.wikimedia.org/T405166#11202712 (10MusikAnimal) >>! In T405166#11200920, @Jdforrester-WMF wrote: >> I'd say for CodeMirror, we need... [19:14:01] (03update) 10brennen: update submodules to recent phorge upstream master (2025-09-08) [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/80 (https://phabricator.wikimedia.org/T404134) [19:19:29] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [19:21:20] 10Phabricator (Upstream): Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245#11202985 (10Peachey88) [19:35:43] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [19:40:54] 10Phabricator (Upstream), 07Upstream: Record in task histories when subscribers are added during a task's creation - https://phabricator.wikimedia.org/T405245#11203030 (10Aklapper) 05Open→03Declined Hi, when there is no timeline entry in the task it means that a value was set at task creation. The curr... [20:12:50] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [20:15:16] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [20:35:12] (03update) 10dduvall: buildkit: Ensure image metadata resolution occurs via buildkitd [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/152 [20:35:15] (03open) 10dduvall: buildkit: Ensure image metadata resolution occurs via buildkitd [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/152 [20:35:19] (03update) 10dduvall: buildkit: Ensure image metadata resolution occurs via buildkitd [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/152 [20:35:20] (03update) 10dduvall: buildkit: Ensure image metadata resolution occurs via buildkitd [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/152 [20:38:26] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] (test/gitlab-runners-staging-ddba) - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [20:38:26] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] (test/gitlab-runners-staging-ddba) - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [20:38:26] (03update) 10dduvall: buildkit: Ensure image metadata resolution occurs via buildkitd [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/153 [20:38:26] (03update) 10dduvall: ci: Testing gitlab-runners-staging (DO NOT MERGE) [repos/releng/blubber] (test/gitlab-runners-staging-ddba) - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/151 [20:38:26] (03open) 10dduvall: buildkit: Ensure image metadata resolution occurs via buildkitd [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/153 [20:38:26] (03close) 10dduvall: buildkit: Ensure image metadata resolution occurs via buildkitd [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/153