[00:02:36] 10Phabricator: Make #tool-docs a subproject of #tools - https://phabricator.wikimedia.org/T308087 (10Samwilson) [00:04:36] 10GitLab, 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10User-brennen: Replicate select published images from GitLab container registry to WMF prod registry - https://phabricator.wikimedia.org/T308080 (10brennen) [00:13:32] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10serviceops, 10Patch-For-Review, 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10brennen) > For "dock... [00:21:50] 10Project-Admins: Make #tool-docs a subproject of #tools - https://phabricator.wikimedia.org/T308087 (10bd808) Retagging per the template task linked from https://www.mediawiki.org/wiki/Phabricator/Project_management#Converting_projects_into_subprojects_/_milestones. https://wikitech.wikimedia.org/wiki/Phabrica... [00:24:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Jenkins: Set up perceptual diffs (visual regression testing) - https://phabricator.wikimedia.org/T64633 (10Jdlrobson) Web team has built out a solution here without jenkins that is working well so far. I am curious Timo if this could s... [00:44:16] 10Project-Admins: Make #tool-docs a subproject of #tools - https://phabricator.wikimedia.org/T308087 (10Aklapper) 05Open→03Resolved a:03Aklapper ` aklapper@phab1001:~$ sudo /srv/phab/phabricator/bin/move_project --subproject --child 5936 --parent 703 --keep-members child Done. ` [01:50:29] 10GitLab: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) [01:51:56] 10GitLab: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn) [03:15:18] 10Project-Admins: Make #tool-docs a subproject of #tools - https://phabricator.wikimedia.org/T308087 (10Samwilson) Thanks! And sorry for the extra work. [07:38:47] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10hashar) [07:52:46] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release Pipeline (Blubber), 10User-dduvall: Blubber must not use easy_install to install pip, but python3-pip - https://phabricator.wikimedia.org/T304860 (10hashar) 05Resolved→03Open Reopening since https://gerrit.wikimedia.org/r/c/mediawiki/serv... [08:03:52] 10GitLab, 10Release-Engineering-Team, 10wikimedia.biterg.io, 10User-AKlapper: How to identify affiliation of indexed GitLab accounts - https://phabricator.wikimedia.org/T306770 (10Aklapper) Would love to have input from folks who better know GitLab here. Currently in Bitergia's Hatstall I only see: `Name... [08:14:15] (03PS14) 10Hashar: Use standalone jobs for Wikibase Selenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/676107 (https://phabricator.wikimedia.org/T287582) [08:20:02] (03PS1) 10Hashar: (DO NOT SUBMIT) inject parameters to wikibase selenium [integration/config] - 10https://gerrit.wikimedia.org/r/790984 [08:35:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10Wikibase (3rd party installations), 10Wikidata, and 3 others: Move some Wikibase selenium tests to a standalone job - https://phabricator.wikimedia.org/T287582 (10hashar) When trying the jobs yesterday they failed again. The reason is th... [08:44:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10hashar) >>! In T225730#7889448, @Tgr wrote: > This is really getting frustrating for... [08:45:21] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10serviceops, 10Kubernetes, 10Patch-For-Review: Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10elukey) I confirm from Aiko's tests that HELM_CACHE_HOME is the prob... [08:52:42] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10serviceops, 10Kubernetes, 10Patch-For-Review: Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10akosiaris) Just to note a semantic thing here: > I am trying to fig... [09:05:47] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10serviceops, 10Kubernetes, 10Patch-For-Review: Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10Joe) >>! In T305729#7920053, @akosiaris wrote: > Just to note a sema... [12:23:50] (03CR) 10Hashar: [C: 04-2] (DO NOT SUBMIT) inject parameters to wikibase selenium [integration/config] - 10https://gerrit.wikimedia.org/r/790984 (owner: 10Hashar) [12:29:31] helping people on the internet and getting a reply stating "You are an absolute legend." :] [12:44:07] (03PS1) 10DCausse: WikibaseCirrusSearch depends on Elastica for phan analysis [integration/config] - 10https://gerrit.wikimedia.org/r/791025 [12:48:49] (03PS1) 10Hashar: [wikimedia/fundraising/crm] add experimental Bullseye job [integration/config] - 10https://gerrit.wikimedia.org/r/791026 (https://phabricator.wikimedia.org/T307178) [12:50:33] (03CR) 10Hashar: [C: 03+2] [wikimedia/fundraising/crm] add experimental Bullseye job [integration/config] - 10https://gerrit.wikimedia.org/r/791026 (https://phabricator.wikimedia.org/T307178) (owner: 10Hashar) [12:52:35] (03Merged) 10jenkins-bot: [wikimedia/fundraising/crm] add experimental Bullseye job [integration/config] - 10https://gerrit.wikimedia.org/r/791026 (https://phabricator.wikimedia.org/T307178) (owner: 10Hashar) [12:56:23] (03PS2) 10DCausse: WikibaseCirrusSearch depends on Elastica for phan analysis [integration/config] - 10https://gerrit.wikimedia.org/r/791025 [13:18:35] 10GitLab, 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Jelto) 05Open→03Resolved a:03Jelto Thanks for opening the task! I can confirm, puma was stuck and can't be stopped which blocked the restore. There were multiple issues: A... [13:31:15] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: CiviCRM CI jobs fails when migrating from Stretch to Bullseye - https://phabricator.wikimedia.org/T307178 (10hashar) I have deployed an experimental job which uses a Bullseye image and triggered it on https://gerrit... [14:10:32] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen, 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual based on Blubber migration - https://phabricator.wikimedia.org/T307538 (10hashar) [14:10:38] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Doing): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10hashar) [14:10:44] 10GitLab, 10Release-Engineering-Team (Doing): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10hashar) [14:18:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS, 10observability, 10cloud-services-team (Kanban): integration instances have lost Diamond/Graphite monitoring - https://phabricator.wikimedia.org/T307655 (10hashar) I have been using the dashboard [[ https://grafana-cloud.wi... [14:18:50] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T305213 (10thcipriani) 05Open→03Resolved [14:21:01] 10GitLab, 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate buildkitd instances as image builders for GitLab - https://phabricator.wikimedia.org/T307810 (10hashar) [14:21:12] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10hashar) [14:21:22] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen, 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual based on Blubber migration - https://phabricator.wikimedia.org/T307538 (10hashar) [14:22:30] 10Release-Engineering-Team (Doing), 10Scap, 10Patch-For-Review: Run scap CI against Stretch, Buster, Bullseye - https://phabricator.wikimedia.org/T308020 (10hashar) [14:23:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul, 10Patch-For-Review: Delete .git/logs on zuul-merger git repo to prevent reflog from being created - https://phabricator.wikimedia.org/T307620 (10hashar) [14:23:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul, 10Patch-For-Review: Delete .git/logs on zuul-merger git repo to prevent reflog from being created - https://phabricator.wikimedia.org/T307620 (10hashar) p:05Triage→03Low a:03hashar [14:42:53] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for bking - https://phabricator.wikimedia.org/T308147 (10bking) [15:03:41] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for bking - https://phabricator.wikimedia.org/T308147 (10RhinosF1) [15:03:57] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for bking - https://phabricator.wikimedia.org/T308147 (10RhinosF1) [15:05:27] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for **YOUR USERNAME** - https://phabricator.wikimedia.org/T308150 (10RhinosF1) [15:05:45] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for **YOUR USERNAME** - https://phabricator.wikimedia.org/T308150 (10RhinosF1) 05Open→03Invalid [15:09:46] 10Phabricator: Tasks created via Phabricator APIs have restricted edit policy - https://phabricator.wikimedia.org/T305998 (10RhinosF1) >>! In T305998#7854480, @Aklapper wrote: > Hmm, the setting that Majavah found has not been changed since 2019 (which makes me assume that nearly all Forms have a non-default set... [15:13:55] 10GitLab, 10Release-Engineering-Team, 10wikimedia.biterg.io, 10User-AKlapper: How to identify affiliation of indexed GitLab accounts - https://phabricator.wikimedia.org/T306770 (10brennen) My first thought here is that username is always LDAP / CAS `uid`, which maps both to email and to groups that might i... [15:14:22] 10GitLab (Misc), 10Release-Engineering-Team, 10wikimedia.biterg.io, 10User-AKlapper: How to identify affiliation of indexed GitLab accounts - https://phabricator.wikimedia.org/T306770 (10brennen) [16:33:39] 10Release-Engineering-Team (Seen), 10User-AKlapper: Update [[mw:Release_Manager_(Wikimedia)]] - https://phabricator.wikimedia.org/T301971 (10thcipriani) 05Open→03Resolved a:03thcipriani I marked myself as release manager since I've been doing the work described on that page. Although I see the translatio... [17:41:47] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops: contint/releases/hosts with helm installed: puppet - Could not find group deployment - https://phabricator.wikimedia.org/T307740 (10Dzahn) This contributes to: ` 17:36 <+icinga-wm> PROBLEM - Ensure hosts are not performing a change on every pupp... [17:52:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul, 10Graphite, 10Upstream: Broken dashboard links on Zuul Status page - https://phabricator.wikimedia.org/T307405 (10hashar) I guess they can be replaced by: https://grafana.wikimedia.org/d/000000321/zuul https://grafana.wikimedia.... [17:52:48] Hello. using pipeline-lib. on https://gerrit.wikimedia.org/r/c/operations/container/miscweb/+/790786 I expected an image to be built and pushed to the docker registry. But that hasn't happened yet. Is that because I made the mistake to merge it.. instead of just doing V+2 and waiting? [17:53:14] can I do something like the 'recheck' to trigger the pipeline build (again)? [17:53:49] mutante: https://integration.wikimedia.org/zuul/ the 'postmerge' build is still running [17:55:21] taavi: ah, I think my confusion was because I already got a V+2 last night. thanks, just waiting a bit [17:58:08] sure enough, I see it published now:) [17:58:46] well, except the link to the registry is 404 but that must be separate [18:58:52] !log gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster [18:58:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:14:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review: Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 (10bking) @Jdforrester-WMF I wasn't sure if the code was in produ... [19:20:42] 10Scap, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add new user identity to Keyholder for scap - https://phabricator.wikimedia.org/T307351 (10RLazarus) 05Open→03Resolved a:03RLazarus This should be all set! ` rzl@deploy1002:~$ sudo run-puppet-agent [...] rzl@deploy1002:~$ sudo keyholder add /... [20:00:16] 10Continuous-Integration-Config, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 3 others: Include WikibaseLexemeCirrusSearch into CirrusSearch tests - https://phabricator.wikimedia.org/T244485 (10Gehel) [21:13:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS, 10observability, 10cloud-services-team (Kanban): Replacement needed for obsolete Diamond/Graphite monitoring of integration instances - https://phabricator.wikimedia.org/T307655 (10bd808) [22:13:12] can't push a change to Gerrit because "Exceeding maximum number of files per change (147582 > 100000)" heh. I knew why I added those files in multiple steps.now just tried to move them around :p [22:29:54] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Create Blubber repo on GitLab, archive Gerrit repo - https://phabricator.wikimedia.org/T307533 (10jeena) [22:30:03] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Implement linting and unit tests for Blubber on GitLab CI - https://phabricator.wikimedia.org/T307534 (10jeena) 05Open→03In progress https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/2 [22:30:34] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10MediaWiki-Core-JobQueue, 10Beta-Cluster-reproducible, and 2 others: Search on betacommons is not indexing anything - https://phabricator.wikimedia.org/T307862 (10EBernhardson) >>! In T307862#7915849, @AlexisJazz wrote: > Some relation with T302699, I thought... [23:20:47] !log gitlab-prod-1001.devtools: container registry currently enabled [23:20:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:31:05] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10serviceops, 10Patch-For-Review, 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) [23:35:08] 10GitLab (Infrastructure), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10serviceops, 10Patch-For-Review, 10cloud-services-team (Kanban): Assess GitLab Container Registry as a default for container build processes - https://phabricator.wikimedia.org/T307537 (10brennen) 05In progress→03Stalled We ena... [23:35:16] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10brennen) [23:35:24] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10brennen) [23:56:25] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10brennen) To capture some earlier discussion with @dduvall and @dancy, I think we've pretty much landed on...