[00:46:03] !log gitlab: disabling container registries on all existing projects (T307537) [00:46:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:46:08] T307537: Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 [01:09:11] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10serviceops, 10Patch-For-Review, 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) [[https://gitlab.wikimedia.org/rep... [01:16:48] 10Project-Admins: Create project tag for Json5Config - https://phabricator.wikimedia.org/T308199 (10Jasonkhanlar) [04:07:51] 10Project-Admins: Create project tag for Json5Config - https://phabricator.wikimedia.org/T308199 (10Aklapper) 05Openβ†’03Stalled Waiting for code and a [documentation page on mediawiki.org](https://www.mediawiki.org/wiki/Extension:Json5Config) to exist; please reset status to `open` once that has happened. [05:01:44] (03PS1) 10Aklapper: doc: Add Developer Portal [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) [05:02:15] (03CR) 10Aklapper: [C: 04-1] "Do not merge yet; Blocked on T261510" [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper) [05:02:52] (03CR) 10Aklapper: [C: 04-1] "Do not merge yet; Blocked on T261510" [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper) [06:48:52] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.39.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T305218 (10Ladsgroup) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/767769 * **Summary**: Changing connection manag... [07:42:07] (03CR) 10Hashar: [C: 03+1] doc: Add Developer Portal [integration/docroot] - 10https://gerrit.wikimedia.org/r/791111 (https://phabricator.wikimedia.org/T302809) (owner: 10Aklapper) [07:43:16] (03CR) 10Hashar: [C: 03+2] WikibaseCirrusSearch depends on Elastica for phan analysis [integration/config] - 10https://gerrit.wikimedia.org/r/791025 (owner: 10DCausse) [07:45:26] (03Merged) 10jenkins-bot: WikibaseCirrusSearch depends on Elastica for phan analysis [integration/config] - 10https://gerrit.wikimedia.org/r/791025 (owner: 10DCausse) [07:53:49] 10Scap, 10SRE, 10SRE-Access-Requests: Add new user identity to Keyholder for scap - https://phabricator.wikimedia.org/T307351 (10jnuche) @RLazarus awesome, thank you so much! I'll verify we can use the new identity once https://gerrit.wikimedia.org/r/c/operations/puppet/+/789146 has been merged. [08:33:43] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate Kaniko as an option to build CI images - https://phabricator.wikimedia.org/T308213 (10jnuche) [09:05:53] 10GitLab: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Jelto) 05Resolvedβ†’03Open Reopening, puma still fails to stop: ` May 12 01:33:19 gitlab2001 gitlab-restore.sh[26219]: timeout: run: puma: (pid 6268) 44600s, want down ` I'll try to find out why thi... [09:34:15] (03Abandoned) 10Hashar: zuul: move doc_destination to a YAML file [integration/config] - 10https://gerrit.wikimedia.org/r/737059 (owner: 10Hashar) [09:41:22] 10Scap, 10SRE, 10SRE-Access-Requests: Add new user identity to Keyholder for scap - https://phabricator.wikimedia.org/T307351 (10jnuche) @RLazarus the new user `scap` exists now on the required hosts, but unfortunately access through Keyholder is not working: ` jnuche@deploy1002:~$ export SSH_AUTH_SOCK=/run... [09:50:01] (03CR) 10Hashar: "CI has to be configured before the additional PipelineLib / Blubber variants added by https://gerrit.wikimedia.org/r/c/mediawiki/tools/sca" [integration/config] - 10https://gerrit.wikimedia.org/r/790747 (https://phabricator.wikimedia.org/T308020) (owner: 10Hashar) [09:50:15] (03Abandoned) 10Hashar: (DO NOT SUBMIT) inject parameters to wikibase selenium [integration/config] - 10https://gerrit.wikimedia.org/r/790984 (owner: 10Hashar) [10:12:41] (03PS1) 10Kosta Harlan: zuul: Add phan dependencies for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/791325 [10:15:10] (03CR) 10Lucas Werkmeister (WMDE): "So this basically makes the whole Wikibase/.phan/stubs directory unnecessary, right? Is there any reason why we wouldn’t want this change?" [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:16:50] (03CR) 10Kosta Harlan: zuul: Add phan dependencies for Wikibase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:17:20] (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] zuul: Add phan dependencies for Wikibase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:17:32] (03CR) 10Kosta Harlan: zuul: Add phan dependencies for Wikibase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:19:07] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10MediaWiki-Core-JobQueue, 10Beta-Cluster-reproducible, 10Discovery-Search (Current work): Search on betacommons is not indexing anything - https://phabricator.wikimedia.org/T307862 (10AlexisJazz) I just uploaded https://commons.wikimedia.beta.wmflabs.org/wi... [10:19:29] (03CR) 10Jakob: [C: 03+1] "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:21:02] (03CR) 10Hashar: [C: 03+2] "Thank you so much for getting rid of stubs ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:23:06] (03Merged) 10jenkins-bot: zuul: Add phan dependencies for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:23:59] (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/791325 (owner: 10Kosta Harlan) [10:29:35] Lucas_WMDE: kostajh: thanks for dropping the phan stubs from Wikibase *fingers crossed* [10:30:45] hashar: is there a task about removing stubs from extensions? [10:31:06] if not, should we make one? [10:33:09] maybe https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Add_phan_to_a_MediaWiki_extension#Advanced should also mention the CI dependencies [10:33:21] (at least it already discourages stubs, which is good) [10:37:28] https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/179112/console didn't work, because my patch mentioned Monolog which I guess is not an extension [10:38:04] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/791326/2/.phan/stubs/monolog.php#b8 [10:39:39] (03PS1) 10Kosta Harlan: zuul: Update phan dependencies for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/791330 [10:40:13] (03PS2) 10Kosta Harlan: zuul: Update phan dependencies for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/791330 [10:41:50] hashar: ^ can you try that please with a recheck on the wikibase patch? The wikibase patch might need some more tweaks [10:43:51] 10Release-Engineering-Team, 10Add-Link, 10Growth-Team, 10Release Pipeline, 10Patch-For-Review: Copy test coverage artifacts from test to coverage variant - https://phabricator.wikimedia.org/T307772 (10kostajh) [10:45:49] kostajh: sure [10:45:59] (03CR) 10Hashar: [C: 03+2] zuul: Update phan dependencies for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/791330 (owner: 10Kosta Harlan) [10:46:32] some of us recently had a discussion about avoiding stubbing mediawiki skins/extensions [10:46:39] I don't think anything got recorded though [10:47:01] something eventually broke in an extension cause phan was not checking [10:47:18] maybe some bits can be added to https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Add_phan_to_a_MediaWiki_extension [10:47:57] (03Merged) 10jenkins-bot: zuul: Update phan dependencies for Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/791330 (owner: 10Kosta Harlan) [10:48:52] deployed [10:52:47] and I did the recheck [11:02:27] Lucas_WMDE: kostajh: Phan still fails https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/179120/console :\ [11:03:29] hm, do we need to add the extensions to the directory_list as in https://www.mediawiki.org/wiki/Continuous_integration/Tutorials/Add_phan_to_a_MediaWiki_extension#Advanced ? [11:03:55] so zuul/parameter_functions.py is solely to have CI to clone the repo [11:04:05] and indeed the Phan config file then has to be amended to refer to those repos [11:04:10] ok, makes sense [11:04:15] so yes [11:04:27] copy paste in both directory_list and exclude_analysis_directory_list [11:04:43] ok, I can do that [11:05:03] there is a task somewhere to have those in `extension.json` so we could then have phan-mediawiki-config to generate those configs [11:05:31] and potentially we could imagine the CI job would be able to do the clone based on whatever the extension has defined (rather than having to update zuul/parameter_functions.py [11:09:25] change updated [11:12:39] oof. sorry for the half-baked patch, doing too many things at once [11:13:47] bah, still failing [11:14:07] https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/179125/console [11:14:33] looks like it needs to be two .. ? [11:14:41] at least that’s what I have in WikibaseQualityConstraints [11:14:45] I’ll try that [11:15:02] oh, of course [11:15:03] right [11:15:04] ../../ [11:15:06] ../../extensions/… [11:15:08] or just ../… [11:15:10] but not ../extensions/ [11:15:16] that doesn’t make sense ^^ [11:15:35] let’s go with ../../extensions/ so it’s more obvious what these paths are [11:15:46] pushed [11:17:57] (03CR) 10Jaime Nuche: [C: 03+1] "LGTM 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/790747 (https://phabricator.wikimedia.org/T308020) (owner: 10Hashar) [11:19:38] now it’s green [11:20:35] hashar: do you want to +2 as both Lucas and I have made changes to that patch? [11:23:29] sure [11:24:09] phan passed https://integration.wikimedia.org/ci/job/mwext-php72-phan-docker/179126/ [11:24:45] I am off for lunch [11:25:48] merci, danke, etc [11:39:16] 10Phabricator: Tasks created via Phabricator APIs have restricted edit policy - https://phabricator.wikimedia.org/T305998 (10Aklapper) Edit Policy of Form 96 is set to author, admins, subscribers, and members of { #Trusted-Contributors | #WMF-NDA }. That makes sense to me. [11:40:47] fun the wiki page refers to the MassMessage extension which has stubs for Echo / Flow https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/MassMessage/%2B/refs/heads/master/.phan/stubs/ [11:46:37] 10Phabricator: Tasks created via Phabricator APIs have restricted edit policy - https://phabricator.wikimedia.org/T305998 (10RhinosF1) >>! In T305998#7924227, @Aklapper wrote: > Edit Policy of Form 96 is set to author, admins, subscribers, and members of { #Trusted-Contributors | #WMF-NDA }. That makes sense to... [11:48:04] 10Phabricator, 10Growth-Team: Creation of a Growth Team Task Form Template - https://phabricator.wikimedia.org/T308234 (10DMburugu) [11:50:03] 10Phabricator, 10Growth-Team: Creation of a Growth Team Task Form Template - https://phabricator.wikimedia.org/T308234 (10DMburugu) [11:59:58] 10Phabricator, 10Growth-Team: Creation of a Growth Team Task Form Template - https://phabricator.wikimedia.org/T308234 (10DMburugu) [12:05:08] 10Phabricator, 10Growth-Team: Creation of a Growth Team Task Form Template - https://phabricator.wikimedia.org/T308234 (10DMburugu) 05Openβ†’03Resolved [12:19:52] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10MediaWiki-Core-JobQueue, 10Beta-Cluster-reproducible, 10Discovery-Search (Current work): Search on betacommons is not indexing anything - https://phabricator.wikimedia.org/T307862 (10dcausse) https://commons.wikimedia.beta.wmflabs.org/wiki/File:Jason_Shaw_... [12:23:23] 10Phabricator, 10Growth-Team: Creation of a Growth Team Task Form Template - https://phabricator.wikimedia.org/T308234 (10Aklapper) @DMburugu: Hi, how was this resolved? Please see https://www.mediawiki.org/wiki/Phabricator/Help/Forms#Creating_custom_forms - thanks! [12:25:27] 10Phabricator: Tasks created via Phabricator APIs have restricted edit policy - https://phabricator.wikimedia.org/T305998 (10Aklapper) not sure if requesting deployment training makes sense if you're not trusted but maybe I'm nitpicking here... [12:29:21] 10Phabricator: Tasks created via Phabricator APIs have restricted edit policy - https://phabricator.wikimedia.org/T305998 (10RhinosF1) >>! In T305998#7924382, @Aklapper wrote: > not sure if requesting deployment training makes sense if you're not trusted but maybe I'm nitpicking here... Maybe not but I'm fairly... [13:31:36] (03PS1) 10Lucas Werkmeister (WMDE): zuul: Add Scribunto to WikibaseLexeme Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) [13:33:33] hashar: another followup to the Phan stubs changes, can you take a look? ^ [13:41:12] (03PS2) 10Lucas Werkmeister (WMDE): zuul: Add Scribunto to more Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) [13:43:34] (03CR) 10jerkins-bot: [V: 04-1] zuul: Add Scribunto to more Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) (owner: 10Lucas Werkmeister (WMDE)) [13:44:26] ugh what a jerk [13:44:46] (/s) [13:46:05] (03PS3) 10Lucas Werkmeister (WMDE): zuul: Add Scribunto to more Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) [13:52:26] 10Release-Engineering-Team, 10MediaWiki-Debug-Logger, 10MediaWiki-Maintenance-system: Maintenance scripts should consistently log errors - https://phabricator.wikimedia.org/T307402 (10Tgr) T308186 was another example of being bitten by this - the maintenance script was failing on some wikis only (some i18n-r... [14:25:35] 10GitLab (Infrastructure): gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10brennen) [14:25:49] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10brennen) [14:26:03] (03PS1) 10BryanDavis: Review access change [wikimedia/developer-portal] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/791257 [14:26:20] 10GitLab (CI & Job Runners), 10serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10brennen) [14:27:24] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10brennen) [14:27:35] (03PS2) 10BryanDavis: Add permissions for TWN integration and release tagging [wikimedia/developer-portal] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/791257 (https://phabricator.wikimedia.org/T308236) [14:27:55] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10User-brennen: Replicate select published images from GitLab container registry to WMF prod registry - https://phabricator.wikimedia.org/T308080 (10brennen) [14:28:15] (03CR) 10BryanDavis: [V: 03+2 C: 03+2] Add permissions for TWN integration and release tagging [wikimedia/developer-portal] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/791257 (https://phabricator.wikimedia.org/T308236) (owner: 10BryanDavis) [14:54:21] (03CR) 10Hashar: [C: 03+2] zuul: Add Scribunto to more Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) (owner: 10Lucas Werkmeister (WMDE)) [14:56:12] (03Merged) 10jenkins-bot: zuul: Add Scribunto to more Phan dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) (owner: 10Lucas Werkmeister (WMDE)) [14:59:06] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) (owner: 10Lucas Werkmeister (WMDE)) [14:59:51] (03CR) 10Lucas Werkmeister (WMDE): zuul: Add Scribunto to more Phan dependencies (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/791375 (https://phabricator.wikimedia.org/T308243) (owner: 10Lucas Werkmeister (WMDE)) [15:05:14] !log gitlab-prod-1001.devtools: soft reboot [15:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:44:30] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/790747 [15:45:09] (03CR) 10Ahmon Dancy: [C: 03+2] [mediawiki/tools/scap] promote experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/790747 (https://phabricator.wikimedia.org/T308020) (owner: 10Hashar) [15:45:45] (03Merged) 10jenkins-bot: [mediawiki/tools/scap] promote experimental jobs [integration/config] - 10https://gerrit.wikimedia.org/r/790747 (https://phabricator.wikimedia.org/T308020) (owner: 10Hashar) [15:46:27] (03CR) 10Ahmon Dancy: "recheck" [tools/scap] - 10https://gerrit.wikimedia.org/r/775400 (https://phabricator.wikimedia.org/T308020) (owner: 10Hashar) [15:53:02] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10serviceops: Need a service account on deploy servers for automated train pre-sync operations - https://phabricator.wikimedia.org/T303857 (10hashar) I am not sure what happened but the `deployment` group does... [16:00:03] !log contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday | https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/ [16:00:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:00:08] could not remember the task for that [16:24:41] (03CR) 10Hashar: [C: 03+1] "Oh nice! thanks for the CI config change deployment." [tools/scap] - 10https://gerrit.wikimedia.org/r/775400 (https://phabricator.wikimedia.org/T308020) (owner: 10Hashar) [16:28:25] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10hashar) [17:12:50] (03PS1) 10Ahmon Dancy: pipeline: target Stretch, Buster and Bullseye [tools/scap] - 10https://gerrit.wikimedia.org/r/791412 (https://phabricator.wikimedia.org/T308020) [17:13:21] (03CR) 10Ahmon Dancy: pipeline: target Stretch, Buster and Bullseye (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/775400 (https://phabricator.wikimedia.org/T308020) (owner: 10Hashar) [17:14:21] (03PS2) 10Ahmon Dancy: pipeline: target Stretch, Buster and Bullseye [tools/scap] - 10https://gerrit.wikimedia.org/r/791412 (https://phabricator.wikimedia.org/T308020) [17:16:41] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10dduvall) 05In progressβ†’03Resolved [17:16:47] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10dduvall) [17:17:45] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10User-brennen: Replicate select published images from GitLab container registry to WMF prod registry - https://phabricator.wikimedia.org/T308080 (10dancy) 05Openβ†’03Declined Not needed if buildkitd's run on t... [17:20:31] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10dduvall) 05In progressβ†’03Resolved The consensus is that buildkitd will provide us more features and c... [17:20:37] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10dduvall) [17:23:22] 10Release-Engineering-Team (GitLab-a-thon 🦊): Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) [17:24:10] 10Release-Engineering-Team (GitLab-a-thon 🦊): Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall) 05Openβ†’03In progress p:05Triageβ†’03High [17:35:53] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10brennen) [17:50:31] 10Release-Engineering-Team (GitLab-a-thon 🦊): Provide GitLab CI include for standard pipeline jobs - https://phabricator.wikimedia.org/T308275 (10dduvall) [18:34:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul, 10Graphite, 10Upstream: Broken dashboard links on Zuul Status page - https://phabricator.wikimedia.org/T307405 (10Krinkle) Yes. [19:14:52] (03PS1) 10Samtar: Zuul: Add Tranve to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/791426 [19:53:09] !log gerrit: triggering full replication to gerrit2001 to test T307137 [19:53:09] . ssh -p 29418 hashar@gerrit.wikimedia.org replication start --all --now --wait --url 'gerrit2@gerrit2001.wikimedia.org' [19:53:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:53:11] T307137: Gerrit replication after a restart takes roughly 5 hours - https://phabricator.wikimedia.org/T307137 [20:08:02] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review: Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 (10bking) I checked `deployment-mediawiki12` and it does indeed l... [20:10:27] 10Gerrit, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Gerrit replication after a restart takes roughly 5 hours - https://phabricator.wikimedia.org/T307137 (10hashar) ` lang=irc 19:57:12 !log Restarting Gerrit ` [20:12:20] 10Gerrit, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Gerrit replication after a restart takes roughly 5 hours - https://phabricator.wikimedia.org/T307137 (10hashar) I first try a manual replication to gerrit2001 but only one thread was processing. I guess the plugin had to be reloaded somehow I... [20:27:28] 10GitLab (Infrastructure), 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) after deploying the change above, now this is what happens when manually starting the unit: ` May 12 19:55:23 gitlab2001 gitlab-restore.sh[17479]: timeout... [20:28:57] 10GitLab (Infrastructure), 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) 05Openβ†’03In progress [20:44:10] (03PS1) 10Brennen Bearnes: backport: add get_user_ssh_env() to gerrit_ssh() [tools/scap] - 10https://gerrit.wikimedia.org/r/791454 [20:44:34] (03CR) 10jerkins-bot: [V: 04-1] backport: add get_user_ssh_env() to gerrit_ssh() [tools/scap] - 10https://gerrit.wikimedia.org/r/791454 (owner: 10Brennen Bearnes) [20:51:45] (03CR) 10SBassett: [C: 03+2] Add new permissions for supplemental security tasks [tools/release] - 10https://gerrit.wikimedia.org/r/787110 (https://phabricator.wikimedia.org/T307036) (owner: 10SBassett) [20:52:30] (03Merged) 10jenkins-bot: Add new permissions for supplemental security tasks [tools/release] - 10https://gerrit.wikimedia.org/r/787110 (https://phabricator.wikimedia.org/T307036) (owner: 10SBassett) [20:52:46] 10Release-Engineering-Team, 10Scap, 10MediaWiki Train Development Environment: train-dev's Gerrit zuul plugin returns a different object than production Gerrit - https://phabricator.wikimedia.org/T308290 (10dancy) [20:58:07] 10Release-Engineering-Team, 10Web Team Visual Regression Framework, 10Readers-Web-Backlog (Needs Prioritization (Tech)): Make visual regression tests run in CI (non-blocking) for the Vector repo - https://phabricator.wikimedia.org/T308194 (10hashar) > For each Gerrit patch set made to Vector, CI runs a comma... [20:58:11] (03CR) 10Jeena Huneidi: [C: 03+1] "Sorry didn't get around to this yet" [tools/scap] - 10https://gerrit.wikimedia.org/r/791454 (owner: 10Brennen Bearnes) [20:59:46] (03CR) 10Ahmon Dancy: "FYI scap CI is broken at the moment. It should be resolved by tomorrow by Antoine and me." [tools/scap] - 10https://gerrit.wikimedia.org/r/791454 (owner: 10Brennen Bearnes) [21:20:40] 10Release-Engineering-Team, 10Scap, 10MediaWiki Train Development Environment: train-dev's Gerrit zuul plugin returns a different object than production Gerrit - https://phabricator.wikimedia.org/T308290 (10dancy) [21:35:42] 10Release-Engineering-Team, 10Scap, 10MediaWiki Train Development Environment: train-dev's Gerrit zuul plugin returns a different object than production Gerrit - https://phabricator.wikimedia.org/T308290 (10jeena) I think we may need to build this plugin from source (https://gerrit.wikimedia.org/r/plugins/zu... [21:41:40] 10Continuous-Integration-Config, 10Platform Engineering, 10Quibble, 10MW-1.38-notes (1.38.0-wmf.23; 2022-02-21), and 2 others: "should mark the revert as a bot edit" test flaky with Apache + php-fpm - https://phabricator.wikimedia.org/T298514 (10Krinkle) Related to {T304766} [22:45:27] when using pipelinelib and CI builds and publishes a docker image, it outputs links to the images on Gerrit. Just that nowadays those links are always 404 [22:46:27] but slightly different URLs with /tags/ get you a list of all of them. I would suggest a fix just don't know where yet [23:58:49] oh interesting! That is likely somewhere in the bowels of pipelinelib.