[00:15:23] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9993226 (10Ladsgroup) That's weird, s1 is enwiki and that's group2. I don't know anything that would do a cross wiki read to enwiki from a group1 wiki... [00:15:53] 10Continuous-Integration-Config, 10MediaWiki-Vendor: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380 (10Krinkle) 03NEW [00:16:01] 10Continuous-Integration-Config, 10MediaWiki-Vendor, 10Quibble: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#9993239 (10Krinkle) [00:31:23] (03PS1) 10Krinkle: Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) [00:31:58] James_F: Reedy oops :) I broke this last year. [00:32:43] I wonder if we had more "X broke all of CI due to a bad composer.json" patch in the last six months, or whether people know not to merge core patches without a Depends-On to a vendor patch. [00:33:13] (03CR) 10CI reject: [V:04-1] Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) (owner: 10Krinkle) [00:33:15] also, if the libs were actualy incompatible, runtime PHPUnit will still fail if the code is covered. but yeah weve been running without lock checks basically [00:36:43] (03PS2) 10Krinkle: Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) [00:37:09] 10Continuous-Integration-Config, 10MediaWiki-Vendor, 10Quibble, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#9993265 (10Krinkle) [00:38:31] (03CR) 10CI reject: [V:04-1] Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) (owner: 10Krinkle) [00:41:55] (03PS3) 10Krinkle: Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) [00:43:46] (03CR) 10CI reject: [V:04-1] Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) (owner: 10Krinkle) [00:50:16] (03PS4) 10Krinkle: Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) [04:43:49] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9993367 (10Marostegui) >>! In T366959#9993226, @Ladsgroup wrote: > That's weird, s1 is enwiki and that's group2. I don't know anything that would do a... [06:26:26] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9993410 (10Aklapper) @ #DBA's: Explicitly asking: Are we fine deploying 1.43.0-wmf.14 to group1 wikis again (lag spike was coincidence time-wise), or... [07:41:54] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Create account for Levi Ferreira on wikimedia.biterg.io - https://phabricator.wikimedia.org/T370089#9993548 (10Aklapper) 05Stalled→03Resolved @Lferreira: The internal support ticket has been resolved, so I assume you can access a b... [07:44:33] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Infrastructure-Foundations, 06SRE: package_builder python-all conflicts with base::standard_packages python2.7 removal - https://phabricator.wikimedia.org/T370337#9993557 (10hashar) `base::standard_packages()` `remove_python2` parameter... [07:48:01] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Cloud-VPS: profile::labs::lvm::srv fails with Invalid argument for --extents: 100%FREE - https://phabricator.wikimedia.org/T370312#9993559 (10hashar) 05In progress→03Resolved I love shooting myself in the foot one year apart and... [07:59:38] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9993586 (10Aklapper) My understanding of comments in T370304 and of the IRC backlog in `#mediawiki_security` is that the train can proceed. [08:20:23] Promoted group1 wikis again to 1.43.0-wmf.14 [08:26:21] !log Deleted Buster based integration-agent-pkgbuilder-1001 integration-agent-pkgbuilder-1002 , replaced them with Bullseye based integration-agent-pkgbuilder-1003 and integration-agent-pkgbuilder-1004 # T360786 [08:26:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:26:24] T360786: Rebuild integration-agent-pkgbuilder-1001 and integration-agent-pkgbuilder-1002 to get rid of Debian Buster - https://phabricator.wikimedia.org/T360786 [08:27:36] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Rebuild integration-agent-pkgbuilder-1001 and integration-agent-pkgbuilder-1002 to get rid of Debian Buster - https://phabricator.wikimedia.org/T360786#9993653 (10hashar) 05Open→03Resolved I have build the two instances using Bullseye... [08:30:53] (03PS1) 10Kevin Bazira: inference-services: add filters for new model-server paths [integration/config] - 10https://gerrit.wikimedia.org/r/1055151 (https://phabricator.wikimedia.org/T369344) [08:31:54] (03CR) 10Kevin Bazira: "Great suggestion, Antoine. Thank you! I have pushed a patch that adds filters for the remaining model-servers: https://gerrit.wikimedia.or" [integration/config] - 10https://gerrit.wikimedia.org/r/1054800 (https://phabricator.wikimedia.org/T369344) (owner: 10Kevin Bazira) [08:32:28] kevinbazira: success ;) [08:32:39] (03CR) 10Hashar: [C:03+2] inference-services: add filters for new model-server paths [integration/config] - 10https://gerrit.wikimedia.org/r/1055151 (https://phabricator.wikimedia.org/T369344) (owner: 10Kevin Bazira) [08:33:30] hashar: super! :D [08:34:16] (03Merged) 10jenkins-bot: inference-services: add filters for new model-server paths [integration/config] - 10https://gerrit.wikimedia.org/r/1055151 (https://phabricator.wikimedia.org/T369344) (owner: 10Kevin Bazira) [08:46:09] kevinbazira: I have deployed the change ;) [08:46:59] hashar: thanks a bunch! :D [09:01:09] !log drop gb_by from globalblocks table in beta cluster [09:01:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:01:40] (03update) 10jnuche: Ask for confirmation instead of aborting if Train task status is no good [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/379 (https://phabricator.wikimedia.org/T369831) (owner: 10aklapper) [09:03:47] (03update) 10jnuche: Ask for confirmation instead of aborting if Train task status is no good [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/379 (https://phabricator.wikimedia.org/T369831) (owner: 10aklapper) [09:09:07] 06Release-Engineering-Team, 07Epic, 07OKR-Work, 13Patch-For-Review: [WE6.2.1] Publish pre-train single version containers - https://phabricator.wikimedia.org/T369115#9993858 (10Aklapper) [09:27:31] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9993905 (10Lucas_Werkmeister_WMDE) {T370396} seems to be a new error in this train, FYI. Not sure if it should be a train blocker. [09:28:50] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9993908 (10Aklapper) Thanks. I've seen that issue but all occurrences so far are on old `1.43.0-wmf.13` (thus not a train blocker) and limited to a si... [09:39:07] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9993952 (10Lucas_Werkmeister_WMDE) Yeah, I just noticed the wrong train version too, sorry for the noise. [09:41:47] hashar: as discussed - T367403#9993970 [09:41:48] T367403: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403 [09:41:58] merci [09:42:21] 06Release-Engineering-Team, 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#9993973 (10Gehel) [09:42:58] gehel: I will try to remember about it and add the ssh client after lunch [09:44:12] We should probably move the installation of the ssh client to the maven image to ensure it si always present when needed [09:51:14] 06Release-Engineering-Team, 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization: Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#9994019 (10Gehel) @brennen : Do you have an opinion on where to host artifacts coming from projects... [09:51:17] 06Release-Engineering-Team, 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization: Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#9994024 (10Gehel) [09:51:23] @brennen question for you ^ [09:52:21] (03CR) 10Itamar Givon: "What do you mean by "imagine the series of patches to work"? Is there anything else we need to do to make this change correctly?" [integration/config] - 10https://gerrit.wikimedia.org/r/1053016 (https://phabricator.wikimedia.org/T369795) (owner: 10Itamar Givon) [10:18:49] 06Release-Engineering-Team, 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#9994109 (10hashar) The images do not have a SSH client by defa... [10:28:29] (03PS1) 10Hashar: dockerfiles: update Java 11 images to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055166 (https://phabricator.wikimedia.org/T335765) [10:29:34] (03CR) 10CI reject: [V:04-1] dockerfiles: update Java 11 images to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055166 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [10:30:39] (03PS2) 10Hashar: dockerfiles: update Java 11 images to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055166 (https://phabricator.wikimedia.org/T335765) [10:35:11] (03PS1) 10Hashar: dockerfiles: add openssh-client to maven-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1055167 (https://phabricator.wikimedia.org/T367403) [10:35:16] (03CR) 10Hashar: [C:03+2] dockerfiles: update Java 11 images to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055166 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [10:36:29] (03Merged) 10jenkins-bot: dockerfiles: update Java 11 images to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055166 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [10:45:57] (03PS2) 10Hashar: dockerfiles: add openssh-client to maven-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1055167 (https://phabricator.wikimedia.org/T367403) [10:45:57] (03PS1) 10Hashar: jjb: switch Java 11 jobs from Buster to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055171 (https://phabricator.wikimedia.org/T335765) [10:48:23] (03CR) 10Hashar: [C:03+2] jjb: switch Java 11 jobs from Buster to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055171 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [10:48:58] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765#9994177 (10hashar) [10:49:04] (03CR) 10Hashar: [C:03+2] dockerfiles: add openssh-client to maven-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1055167 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [10:49:33] (03Merged) 10jenkins-bot: jjb: switch Java 11 jobs from Buster to Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1055171 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [10:50:17] (03Merged) 10jenkins-bot: dockerfiles: add openssh-client to maven-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1055167 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [10:52:08] (03PS1) 10Hashar: jjb: update Java 11 jobs to get openssh-client [integration/config] - 10https://gerrit.wikimedia.org/r/1055173 (https://phabricator.wikimedia.org/T367403) [10:53:38] (03PS2) 10Hashar: jjb: update Java 11 jobs to get openssh-client [integration/config] - 10https://gerrit.wikimedia.org/r/1055173 (https://phabricator.wikimedia.org/T367403) [10:53:38] (03PS1) 10Hashar: dockerfiles: move file from java11 to maven-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1055174 [10:53:57] (03CR) 10Hashar: [C:03+2] dockerfiles: move file from java11 to maven-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1055174 (owner: 10Hashar) [10:54:09] (03CR) 10Hashar: [C:03+2] jjb: update Java 11 jobs to get openssh-client [integration/config] - 10https://gerrit.wikimedia.org/r/1055173 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [10:55:35] (03Merged) 10jenkins-bot: dockerfiles: move file from java11 to maven-java11 [integration/config] - 10https://gerrit.wikimedia.org/r/1055174 (owner: 10Hashar) [10:55:51] (03Merged) 10jenkins-bot: jjb: update Java 11 jobs to get openssh-client [integration/config] - 10https://gerrit.wikimedia.org/r/1055173 (https://phabricator.wikimedia.org/T367403) (owner: 10Hashar) [10:57:27] 06Release-Engineering-Team, 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, and 2 others: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#9994191 (10hashar) The ssh client is now available in the image `docker-registry... [11:27:43] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Rebuild integration-agent-pkgbuilder-1001 and integration-agent-pkgbuilder-1002 to get rid of Debian Buster - https://phabricator.wikimedia.org/T360786#9994245 (10Jdforrester-WMF) Should we delete the old ones to close this out? [11:30:23] (03PS1) 10Arthur taylor: Run Browser tests in parallel if QUIBBLE_SELENIUM_PARALLEL is set [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055185 (https://phabricator.wikimedia.org/T370033) [12:14:45] (03PS2) 10Daniel Kinzler: extension_survey: add mode for collection service usage [tools/code-utils] - 10https://gerrit.wikimedia.org/r/1054478 [12:16:36] (03CR) 10Daniel Kinzler: Introduce extension_survey.py (031 comment) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/1054308 (owner: 10Daniel Kinzler) [12:21:56] hashar: could use a second pair of eyes at https://gerrit.wikimedia.org/r/1054966 for T370380 [12:21:56] T370380: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380 [12:22:09] Maybe I'm misunderstanding it. [12:41:10] it is in my backlog yes ;) [12:41:31] and something else is broken with `quibble --change` which Arthur from WMDE mentioned to me yesterday [12:41:48] there is a setup_environment() method which is severly broken ;) [12:43:23] (03CR) 10Hashar: [C:03+2] Update Query Builder test runners to node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1053016 (https://phabricator.wikimedia.org/T369795) (owner: 10Itamar Givon) [12:44:29] (03CR) 10CI reject: [V:04-1] Update Query Builder test runners to node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1053016 (https://phabricator.wikimedia.org/T369795) (owner: 10Itamar Givon) [12:45:42] (03CR) 10Hashar: [C:03+2] "It failed because I did not create the `branchdeploy-wikidata-query-builder-node20-npm` job in Jenkins ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/1053016 (https://phabricator.wikimedia.org/T369795) (owner: 10Itamar Givon) [12:46:52] (03Merged) 10jenkins-bot: Update Query Builder test runners to node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1053016 (https://phabricator.wikimedia.org/T369795) (owner: 10Itamar Givon) [12:51:03] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/PageCreationNotif] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053952 (https://phabricator.wikimedia.org/T367673) (owner: 10Pppery) [12:51:17] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/StickToThatLanguage] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053958 (https://phabricator.wikimedia.org/T367670) (owner: 10Pppery) [12:51:25] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/DeleteOwn] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053959 (https://phabricator.wikimedia.org/T366663) (owner: 10Pppery) [12:51:35] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/CollaborationKit] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053960 (https://phabricator.wikimedia.org/T368092) (owner: 10Pppery) [12:52:11] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageCreationNotif] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053952 (https://phabricator.wikimedia.org/T367673) (owner: 10Pppery) [12:52:22] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/StickToThatLanguage] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053958 (https://phabricator.wikimedia.org/T367670) (owner: 10Pppery) [12:52:31] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/DeleteOwn] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053959 (https://phabricator.wikimedia.org/T366663) (owner: 10Pppery) [12:52:43] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CollaborationKit] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1053960 (https://phabricator.wikimedia.org/T368092) (owner: 10Pppery) [12:52:56] !log Zuul: [mediawiki/extensions/PageCreationNotif] Mark as archived, for T367673 [12:52:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:53:00] T367673: Archive the PageCreationNotif extension - https://phabricator.wikimedia.org/T367673 [12:53:19] !log Zuul: [mediawiki/extensions/StickToThatLanguage] Mark as archived, for T367670 [12:53:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:53:21] T367670: Archive the Stick To That Language extension - https://phabricator.wikimedia.org/T367670 [12:55:20] !log Zuul: [mediawiki/extensions/DeleteOwn] Mark as archived, for T366663 [12:55:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:55:23] T366663: Archive the DeleteOwn extension - https://phabricator.wikimedia.org/T366663 [12:56:01] !log Zuul: [mediawiki/extensions/CollaborationKit] Mark as archived, for T368092 [12:56:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:56:04] T368092: Archive the CollaborationKit extension - https://phabricator.wikimedia.org/T368092 [13:18:57] Krinkle: so Quibble / MW_SKIP_EXTERNAL_DEPENDENCIES it is interesting you bring this up today [13:19:25] I noticed use_vendor might be broken yesterday while I investigated why `quibble --change` does not work anymore [13:19:43] I guess it is a coincidence [13:20:03] I noticed it was passing a positional to a named arg later in the list [13:20:05] that looked sus to me [13:20:16] but I'm not python native enough to know that is' actualy broken or just looks weird [13:20:23] - workspace, mw_install_path, log_dir, tmp_dir, zuul_env [13:20:26] yeah [13:20:29] + [13:20:37] that is the one that also broke `quibble --change` [13:20:39] + workspace, mw_install_path, log_dir, tmp_dir, zuul_env=zuul_env [13:20:47] haha, yeah so I found that in CR on myself, but didn't reproduce hte ug [13:20:50] the bug* [13:21:32] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9994648 (10Marostegui) >>! In T366959#9993586, @Aklapper wrote: > My understanding of comments in T370304 and of the IRC backlog in `#mediawiki_securi... [13:21:39] the thing is that `_setup_environment` 5th argument is use_vendor and the 6th argument is zuul_env [13:21:59] but the call pass zuul_env has the 5th argument [13:22:05] s/has/as/ [13:22:27] so the feature flags are mixed up [13:22:48] and if you want to have `MW_SKIP_EXTERNAL_DEPENDENCIES` set, you can thus use `quibble --change` [13:23:25] and I don't understand how I broke it [13:25:28] https://gerrit.wikimedia.org/r/c/integration/quibble/+/846812/3..4 [13:25:30] rebased [13:25:42] which introduced the use_vendor flag [13:25:53] but I did not adjust the call to setup_environment [13:26:00] and there is no tests covering the feats [13:26:02] easy [13:26:05] Aha. [13:26:23] so that is a case of pressing `[REBASE]` and not writing tests breaking stuff [13:26:30] that is working as expected [13:26:53] PEBCAK [13:27:41] what I dont get is why we all discover this bug at the same time [13:28:00] when I broke it back in september [14:07:09] 06Release-Engineering-Team, 06collaboration-services: Create test Gerrit instance - https://phabricator.wikimedia.org/T363196#9994848 (10BCornwall) FYI, the test instance isn't a priority for me any more, though it would be nice to have eventually! [14:13:00] it is even worse [14:13:13] self._setup_environment() is called twice [14:13:59] due to the rebase [14:14:00] fun [14:31:50] (03PS1) 10Hashar: Fix --change which never worked [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 [14:32:49] (03CR) 10Hashar: Apply patches in `/cache/patches` during Quibble plan execution (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054492 (https://phabricator.wikimedia.org/T370124) (owner: 10Arthur taylor) [14:36:05] (03CR) 10Hashar: [C:04-1] Apply patches in `/cache/patches` during Quibble plan execution [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054492 (https://phabricator.wikimedia.org/T370124) (owner: 10Arthur taylor) [14:40:02] (03PS5) 10Hashar: Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) (owner: 10Krinkle) [14:42:47] (03CR) 10Hashar: [C:03+2] "Passing zuul_env using a keyword argument has fixed `quibble --change` and since that is unrelated to fixing MW_SKIP_EXTERNAL_DEPENDENCIES" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) (owner: 10Krinkle) [15:02:38] (03approved) 10dancy: Ask for confirmation instead of aborting if Train task status is no good [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/379 (https://phabricator.wikimedia.org/T369831) (owner: 10aklapper) [15:02:49] (03merge) 10dancy: Ask for confirmation instead of aborting if Train task status is no good [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/379 (https://phabricator.wikimedia.org/T369831) (owner: 10aklapper) [15:03:09] 06Release-Engineering-Team, 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization: Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#9995143 (10Ottomata) @gehel, would it be easier if projects are hosted in GitLab? We are planning... [15:14:59] 06Release-Engineering-Team, 06Data-Platform-SRE, 10Dumps-Generation, 10MW-on-K8s, 06serviceops: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#9995235 (10VirginiaPoundstone) @Joe thanks for the ping. Just to keep you posted about our prog... [15:15:48] 06Release-Engineering-Team, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#9995239 (10VirginiaPoundstone) [15:16:28] 06Release-Engineering-Team: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427 (10ItamarWMDE) 03NEW [15:17:08] 06Release-Engineering-Team, 10wmde-wikidata-tech: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427#9995267 (10ItamarWMDE) [15:23:34] (03PS1) 10Pppery: Review access change [phabricator/translations] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1055251 [15:24:03] (03PS2) 10Pppery: Allow Jenkins-bot to submit changes [phabricator/translations] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1055251 [15:29:18] 10Phabricator (Upstream), 10Wikimedia-Phabricator-Extensions, 07Upstream: Address FIXMEs in Phabricator-translations - https://phabricator.wikimedia.org/T365853#9995341 (10Pppery) 05Open→03Resolved Confirming `grep -R FIXME projects` returns no results. [15:29:53] 10Phabricator (Upstream), 10Wikimedia-Phabricator-Extensions, 07Upstream: Address FIXMEs in Phabricator-translations - https://phabricator.wikimedia.org/T365853#9995345 (10Pppery) (For visibility as this involved both upstream and downstream changes) [15:33:20] (03open) 10btullis: Add ferretdb and growthbook data-engineering projects to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92 (https://phabricator.wikimedia.org/T365839) [15:42:03] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10SonarQube Bot, 07Code-Health, 10Quality-and-Test-Engineering-Team (SonarCloud Admin): Improve speed of codehealth checks - https://phabricator.wikimedia.org/T351561#9995423 (10pwangai) 05Open→03In progress a:03pwangai [15:46:48] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995463 (10hubaishan) Can you skip arwiki from this Update. Arabic month names changed and this may stop some bots from run there [15:46:49] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10wmde-wikidata-tech, 07ci-test-error: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427#9995460 (10hashar) sharp installs libvips and the code defines the cache using: ` lang=javascript const cachePath = func... [15:51:48] (03update) 10btullis: Add ferretdb and growthbook data-engineering projects to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92 (https://phabricator.wikimedia.org/T365839) [15:53:27] (03update) 10btullis: Add ferretdb and growthbook data-engineering projects to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92 (https://phabricator.wikimedia.org/T365839) [15:55:01] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995482 (10Pppery) That's not how train deployments work. If there's a specific patch you don't want to roll out to all wikis you could potentially re... [15:55:40] (03update) 10btullis: Add ferretdb and growthbook data-engineering projects to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92 (https://phabricator.wikimedia.org/T365839) [16:01:14] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10wmde-wikidata-tech, 07ci-test-error: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427#9995514 (10hashar) After digging into the npm code I eventually found the mixed cased comes from the Docker images!!! `... [16:02:02] (03merge) 10dancy: Add ferretdb and growthbook data-engineering projects to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92 (https://phabricator.wikimedia.org/T365839) (owner: 10btullis) [16:02:05] (03update) 10dancy: Add ferretdb and growthbook data-engineering projects to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/92 (https://phabricator.wikimedia.org/T365839) (owner: 10btullis) [16:05:01] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995536 (10hubaishan) >>! In T366959#9995482, @Pppery wrote: > That's not how train deployments work. If there's a specific patch you don't want to ro... [16:05:54] 10MediaWiki-Releasing, 10Release-Engineering-Team (Seen), 10MediaWiki-General: Use semantic(-ish) versioning for MediaWiki by dropping the "1." part of the version number - https://phabricator.wikimedia.org/T336128#9995537 (10Izno) I think adopting calendar versioning makes generally more sense with how Medi... [16:08:40] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995540 (10Pppery) Any changes you made on translatewiki today won't get deployed until next train. And backporting i18n changes is apparently difficu... [16:11:25] (03update) 10dancy: deploy_promote.py: Revise commit message summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/383 (https://phabricator.wikimedia.org/T370220) [16:15:52] 10Release-Engineering-Team (Seen), 10Data Pipelines, 06Data-Engineering, 10Data-Platform-SRE (2024.07.08 - 2024.07.28), 13Patch-For-Review: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#9995566 (10Ottomata) [16:16:15] (03update) 10dancy: deploy_promote.py: Revise commit message summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/383 (https://phabricator.wikimedia.org/T370220) [16:16:43] (03update) 10dancy: utils.py: read_wikiversions handle FORCE_MW_VERSION [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/382 (https://phabricator.wikimedia.org/T369115) [16:19:44] (03PS1) 10Hashar: dockerfiles: change NPM_CONFIG_cache to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) [16:20:10] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10wmde-wikidata-tech, 07ci-test-error, 13Patch-For-Review: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427#9995577 (10hashar) a:03hashar [16:20:52] (03update) 10dancy: utils.py: read_wikiversions handle FORCE_MW_VERSION [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/382 (https://phabricator.wikimedia.org/T369115) [16:21:36] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995598 (10thcipriani) >>! In T366959#9995535, @hubaishan wrote: >>>! In T366959#9995482, @Pppery wrote: >> That's not how train deployments work. If... [16:22:53] 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10VPS-project-devtools: Puppet failure on deploy-1006.devtools.eqiad1.wikimedia.cloud - Not authorized to call search on /file_metadata/volatile/GeoIP - https://phabricator.wikimedia.org/T370436 (10brennen) 03NEW [16:24:17] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995630 (10Pppery) The specific changes being complained about seem to be https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1054194/1/languages/i18n/a... [16:25:37] (03update) 10dancy: utils.py: read_wikiversions handle FORCE_MW_VERSION [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/382 (https://phabricator.wikimedia.org/T369115) [16:27:15] (03merge) 10dancy: utils.py: read_wikiversions handle FORCE_MW_VERSION [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/382 (https://phabricator.wikimedia.org/T369115) [16:28:20] 06Release-Engineering-Team, 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, 10Data-Engineering (Q1 2024 July 1st - September 30th): Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#9995650 (10lbowmaker) [16:30:51] (03CR) 10Hashar: "Adding James and Timo as reviewers since I know they know more about npm configuration than me :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [16:34:58] (03update) 10thcipriani: deploy_promote.py: Revise commit message summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/383 (https://phabricator.wikimedia.org/T370220) (owner: 10dancy) [16:45:49] (03update) 10dancy: deploy_promote.py: Revise commit message summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/383 (https://phabricator.wikimedia.org/T370220) [16:45:54] (03merge) 10dancy: deploy_promote.py: Revise commit message summary [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/383 (https://phabricator.wikimedia.org/T370220) [16:48:51] (03CR) 10Krinkle: dockerfiles: change NPM_CONFIG_cache to all lower case (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [16:50:11] (03CR) 10Krinkle: dockerfiles: change NPM_CONFIG_cache to all lower case (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [16:58:00] (03open) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [16:58:05] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [16:58:12] upstream nodejs 22.5 broke the abiluty to run `npm install` :P https://github.com/npm/cli/issues/7657 [16:58:32] github crying out loud because of our auto-upgrading / wildcard "nodejs 22" images adopting it everywhere as soon as it came out [16:58:52] e.g. random comment/doc fixes failing CI because eslint didn't get installed https://github.com/qunitjs/qunit/actions/runs/9995491014/job/27627747066 [16:59:05] and it's doing it without a non-zero exit code so it fails with random erorrs like eslint not found [16:59:19] (if you expand the 'npm ci' step you see the install failure) [17:01:04] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [17:04:09] (03open) 10dancy: scap mediawiki-status command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/385 [17:04:12] (03update) 10dancy: scap mediawiki-status command [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/385 [17:04:27] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [17:04:28] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I7c40c89100420712bb944ccd78380599ac376928) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [17:04:36] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [17:04:37] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I7c40c89100420712bb944ccd78380599ac376928) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [17:06:09] @dancy we have a train blocker [17:06:20] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995793 (10Jdlrobson) Hi we just discovered a train blocker that impacts dark mode. We're creating a bug and working on a patch now but please don't r... [17:06:26] I noted on ticket but pinging here in case you miss it. I see you are cutting the train now? [17:06:37] I'm not doing any train operations at the moment. [17:06:42] ok cool [17:06:44] Thanks for the report. I'll take a look. [17:06:46] hopefully a fix will be up in the hour [17:06:54] Perfect! [17:07:09] Normal train window starts in an hour so that works nicely. [17:11:46] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I7c40c89100420712bb944ccd78380599ac376928) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [17:14:28] (03open) 10catrope: Build the Codex docs again [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/622 [17:16:05] 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10VPS-project-devtools: Puppet failure on deploy-1006.devtools.eqiad1.wikimedia.cloud - Not authorized to call search on /file_metadata/volatile/GeoIP - https://phabricator.wikimedia.org/T370436#9995839 (10Dzahn) I have started invest... [17:16:25] (03open) 10catrope: Copy all Codex files, don't try to list them all [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/623 [17:18:30] (03update) 10catrope: Copy all Codex files, don't try to list them all [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/623 [17:18:32] (03update) 10catrope: Copy all Codex files, don't try to list them all [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/623 [17:24:01] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9995898 (10Jdlrobson) [17:41:19] (03open) 10catrope: Add Chart extension [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/624 (https://phabricator.wikimedia.org/T369943) [18:13:10] (03update) 10jforrester: Add Chart extension to production release list [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/86 [18:13:18] (03approved) 10jforrester: Add Chart extension to production release list [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/86 [18:14:15] (03merge) 10jforrester: Add Chart extension to production release list [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/86 [18:24:29] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#9996177 (10Scott_French) `appservers-ro.discovery.wmnet` and `api-ro.discovery.wmnet` now resolve to failoid, by way of manually upda... [18:40:48] thcipriani: Southparkfan has offered to help with some deployment-prep upgrades. Have you two met? [18:49:36] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.43.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T366959#9996348 (10Pppery) Splitting the Arabic month issue out to T370456 [18:51:14] (03update) 10matmarex: Build the Codex docs again [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/622 (owner: 10catrope) [18:56:55] dancy: i'm still getting review, but close to having that patch merged and a cherry pick up [18:57:18] 👍🏾 [19:01:32] (03open) 10dancy: Release 4.93.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/386 [19:02:49] (03merge) 10dancy: Release 4.93.0-1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/386 [19:03:19] we're just waiting for the parent patch to merge, but https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/1055279 will need to go out with the train dancy [19:04:06] OK. I'll backport it before rolling the train today. [19:04:37] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458 (10Andrew) 03NEW [19:04:42] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-push-notifications01.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370459 (10Andrew) 03NEW [19:04:52] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460 (10Andrew) 03NEW [19:05:06] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461 (10Andrew) 03NEW [19:05:24] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-shellbox.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370462 (10Andrew) 03NEW [19:06:43] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370465 (10Andrew) 03NEW [19:06:45] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-urldownloader03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370466 (10Andrew) 03NEW [19:06:48] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370467 (10Andrew) 03NEW [19:09:21] (03merge) 10matmarex: Copy all Codex files, don't try to list them all [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/623 (owner: 10catrope) [19:10:11] andrewbogott: no! I've seen Southparkfan around, but we've not met. Howdy Southparkfan o/ [19:10:35] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation), 13Patch-For-Review: Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9996518 (10Andrew) [19:11:13] thcipriani: SPF has done some security-adjacent work elsewhere in cloud-vps so I don't have any reservations about him security-wise. Will you to add him to the project so he can scout around? [19:12:41] +1 for SPF [19:15:20] Also +1 for SPF [19:25:59] (03update) 10matmarex: Add Chart extension [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/624 (https://phabricator.wikimedia.org/T369943) (owner: 10catrope) [19:28:13] (03merge) 10matmarex: Add Chart extension [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/624 (https://phabricator.wikimedia.org/T369943) (owner: 10catrope) [19:30:28] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370465#9996597 (10Andrew) [19:30:58] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370467#9996612 (10Andrew) [19:31:08] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-urldownloader03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370466#9996614 (10Andrew) [19:31:59] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-sessionstore04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370461#9996621 (10Andrew) [19:32:04] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370460#9996625 (10Andrew) [19:32:11] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-push-notifications01.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370459#9996628 (10Andrew) [19:32:22] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace poolcounter06.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370458#9996629 (10Andrew) [19:32:30] I'll add SPF as an admin [19:34:51] (03PS1) 10Dreamy Jazz: Zuul: Add GlobalBlocking as a dependency for CheckUser [integration/config] - 10https://gerrit.wikimedia.org/r/1055284 (https://phabricator.wikimedia.org/T370463) [19:39:42] (03PS2) 10Dreamy Jazz: Zuul: [mediawiki/extensions/CheckUser] Add GlobalBlocking dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055284 (https://phabricator.wikimedia.org/T370463) [19:40:06] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9996645 (10Andrew) [19:42:42] {{done}} SPF should now have all the access they need for deployment-prep [19:49:57] 10Phabricator, 06Release-Engineering-Team, 10Phatality: Phatality submit button gives HTTP 403 error ("Forbidden You don't have permission to access this resource.") - https://phabricator.wikimedia.org/T370160#9996691 (10brennen) 05Open→03Resolved a:03brennen Apache configuration issue; should now... [19:57:27] @dancy thanks! should be good to go! [19:58:17] ok.. starting the process [20:02:49] (03update) 10thcipriani: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 (owner: 10dancy) [20:03:11] 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10VPS-project-devtools, 13Patch-For-Review: Puppet failure on deploy-1006.devtools.eqiad1.wikimedia.cloud - Not authorized to call search on /file_metadata/volatile/GeoIP - https://phabricator.wikimedia.org/T370436#9996722 (10Dzahn)... [20:03:40] (03approved) 10thcipriani: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 (owner: 10dancy) [20:04:32] 10Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 10VPS-project-devtools, 13Patch-For-Review: Puppet failure on deploy-1006.devtools.eqiad1.wikimedia.cloud - Not authorized to call search on /file_metadata/volatile/GeoIP - https://phabricator.wikimedia.org/T370436#9996724 (10Dzahn)... [20:06:12] (03open) 10matmarex: Replace custom attempts at monitoring with status dashboard [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/625 (https://phabricator.wikimedia.org/T369535) [20:06:32] brennen: puppet on deploy-1006 fixed (finally):)! [20:06:51] mutante: thanks! [20:07:08] (03merge) 10matmarex: Replace custom attempts at monitoring with status dashboard [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/625 (https://phabricator.wikimedia.org/T369535) [20:08:05] brennen: it actually means "scap deploys in cloud VPS now possible" in general, arguably :) [20:08:19] (with bookworm) [20:16:42] nice. [20:26:26] Jdlrobson: Looks like some tests failed: https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/19215/console [20:27:57] dancy: doesn't look related... [20:28:13] Can we try again or force merge? [20:28:25] O [20:28:32] I'm getting an error with a flagged revisions test on master [20:30:00] ok.. trying to force it thruogh [20:30:03] *through [20:30:13] Looks like our CI might be playing up.. I'll investigate and raise a bug [20:33:07] Jdlrobson: Did you find an error message in that console output? [20:33:16] https://phabricator.wikimedia.org/T370476 [20:33:23] (for master) - might be related [20:34:11] bleh. Broken CI is bad for backports [20:34:13] https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74/19215/console looks like a Wikibase failure? [20:35:35] I have no idea how to find out what's an error message in those logs [20:35:40] It's overwhelming [20:35:59] * dancy thinks search for "error" and laughs. [20:36:04] what was the download location of "sshecret" ? [20:37:02] (03PS1) 10Dreamy Jazz: Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055291 (https://phabricator.wikimedia.org/T369861) [20:37:24] (03CR) 10Krinkle: [C:03+2] Fix --change which never worked [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 (owner: 10Hashar) [20:38:53] (03PS2) 10Dreamy Jazz: Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055291 (https://phabricator.wikimedia.org/T369861) [20:47:15] dancy: i can confirm the fix on the debug servers [20:47:25] yay! [20:50:26] (03CR) 10Tchanders: [C:03+1] Zuul: [mediawiki/extensions/CheckUser] Add GlobalBlocking dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055284 (https://phabricator.wikimedia.org/T370463) (owner: 10Dreamy Jazz) [20:53:35] (03CR) 10CI reject: [V:04-1] Fix --change which never worked [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 (owner: 10Hashar) [20:53:36] (03CR) 10CI reject: [V:04-1] Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) (owner: 10Krinkle) [21:03:16] (03CR) 10Tchanders: [C:03+1] Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055291 (https://phabricator.wikimedia.org/T369861) (owner: 10Dreamy Jazz) [21:15:27] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370467#9996910 (10andrea.denisse) The VM can be safely removed as XHGui is now s... [21:23:38] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370467#9996923 (10andrea.denisse) I tried to delete the instance but I was unabl... [21:29:11] (03open) 10dancy: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 [21:29:11] (03update) 10dancy: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 [21:29:12] (03open) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] (master-Iffb48120bce3185bf7e7c17d6810cedad7dc11d5) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/388 [21:29:12] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] (master-Iffb48120bce3185bf7e7c17d6810cedad7dc11d5) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/388 [21:29:13] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I9ba35e6c4e74dddb36fc7acdc1a5b5ba969ba396) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [21:29:17] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I9ba35e6c4e74dddb36fc7acdc1a5b5ba969ba396) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [21:29:21] (03update) 10dancy: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 [21:29:25] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] (master-Iffb48120bce3185bf7e7c17d6810cedad7dc11d5) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/388 [21:30:49] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-xhgui03.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370467#9996940 (10Andrew) 05Open→03Resolved a:03Andrew You're right! S... [21:32:11] (03update) 10dancy: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 [21:32:12] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] (master-Iffb48120bce3185bf7e7c17d6810cedad7dc11d5) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [21:32:13] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I7c40c89100420712bb944ccd78380599ac376928) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [21:32:21] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] (master-Iffb48120bce3185bf7e7c17d6810cedad7dc11d5) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [21:32:21] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I7c40c89100420712bb944ccd78380599ac376928) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [21:32:24] (03close) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] (master-Iffb48120bce3185bf7e7c17d6810cedad7dc11d5) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/388 [21:32:28] (03update) 10dancy: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 [21:35:14] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] (master-Iffb48120bce3185bf7e7c17d6810cedad7dc11d5) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [21:35:14] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] (master-I7c40c89100420712bb944ccd78380599ac376928) - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [21:35:18] (03update) 10dancy: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 [21:35:42] thcipriani: o/ hi! Thanks for adding :-) [21:37:06] (03update) 10dancy: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 [21:37:27] nice to meet you, hopefully I can be of help to the -prep upgrades - let me know if you would like to have a short conversation/meeting beforehand [21:43:42] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-ircd02 with a Bullseye or Bookworm host - https://phabricator.wikimedia.org/T369919#9996975 (10Southparkfan) a:03Southparkfan [22:13:53] (03approved) 10thcipriani: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 (owner: 10dancy) [22:14:22] (03merge) 10thcipriani: prep.py: _select_reference_directory() improvements [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/387 (owner: 10dancy) [22:14:25] (03update) 10dancy: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 [22:20:49] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-ircd02 with a Bullseye or Bookworm host - https://phabricator.wikimedia.org/T369919#9997107 (10Southparkfan) `role::mw_rc_irc` seems to work fine on a Bullseye box, except for a loop. Puppet run... [22:26:00] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-ircd02 with a Bullseye or Bookworm host - https://phabricator.wikimedia.org/T369919#9997111 (10Southparkfan) Loop fixed by setting `profile::base::remove_python2_on_bullseye: false` on prefix lev... [22:26:07] (03merge) 10thcipriani: utils.py: parse_wmf_version: Handle branch_cut_pretest [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/384 (owner: 10dancy) [22:26:10] (03update) 10dancy: utils.py: Handle "next" version [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) [22:42:52] (03approved) 10thcipriani: utils.py: Handle "next" version [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) (owner: 10dancy) [22:42:56] (03merge) 10thcipriani: utils.py: Handle "next" version [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/381 (https://phabricator.wikimedia.org/T369112) (owner: 10dancy) [23:22:23] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Rebuild or delete deployment-docker-changeprop01 - https://phabricator.wikimedia.org/T369913#9997194 (10Southparkfan) Instance is offline, seems to be superseded by `deployment-changeprop-1.deployment-prep.eqiad1.w... [23:27:21] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-jobrunner04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370487 (10Southparkfan) 03NEW [23:27:25] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Remove or replace deployment-jobrunner04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370487#9997223 (10Southparkfan) a:03Southparkfan [23:29:02] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9997226 (10Southparkfan)