[06:21:40] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/Wikistories] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/895333 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai)
[06:21:42] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/Gadgets] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/895329 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai)
[06:22:54] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikistories] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/895333 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai)
[06:22:57] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Gadgets] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/895329 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai)
[06:23:56] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895329 and https://gerrit.wikimedia.org/r/c/integration/config/+/895333 | T321837
[06:24:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[06:24:01] <stashbot>	 T321837: Repositories integrated into Codehealth Pipeline (Production) - https://phabricator.wikimedia.org/T321837
[06:25:08] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: rename Quibble fullrun job [integration/config] - 10https://gerrit.wikimedia.org/r/895142 (owner: 10Hashar)
[06:26:08] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: use a job template for Quibble fullrun jobs [integration/config] - 10https://gerrit.wikimedia.org/r/895143 (owner: 10Hashar)
[06:26:18] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: rename Quibble fullrun job [integration/config] - 10https://gerrit.wikimedia.org/r/895142 (owner: 10Hashar)
[06:26:53] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895142
[06:26:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[06:27:18] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: use a job template for Quibble fullrun jobs [integration/config] - 10https://gerrit.wikimedia.org/r/895143 (owner: 10Hashar)
[06:48:11] <wikibugs>	 10Phabricator, 10DBA: Switchover m3 master db1101 -> db1159 - https://phabricator.wikimedia.org/T331387 (10Marostegui)
[06:53:07] <wikibugs>	 10Phabricator, 10DBA: Switchover m3 master db1101 -> db1159 - https://phabricator.wikimedia.org/T331387 (10Marostegui)
[06:54:40] <wikibugs>	 10Phabricator, 10DBA: Switchover m3 master db1101 -> db1159 - https://phabricator.wikimedia.org/T331387 (10Marostegui) Test
[06:55:43] <wikibugs>	 10Phabricator, 10DBA: Switchover m3 master db1101 -> db1159 - https://phabricator.wikimedia.org/T331387 (10Marostegui)
[06:56:19] <wikibugs>	 10Phabricator, 10DBA: Switchover m3 master db1101 -> db1159 - https://phabricator.wikimedia.org/T331387 (10Marostegui) 05Open→03Resolved All done
[07:38:48] <elukey>	 hi folks!
[07:39:00] <elukey>	 I am getting a weird issue from CI: https://integration.wikimedia.org/ci/job/inference-services-pipeline-outlink/296/console
[07:39:17] <elukey>	 "expected to call org.wikimedia.integration.PipelineRunner.<init> but wound up catching org.wikimedia.integration.Utility.parseImageRef; see: https://jenkins.io/redirect/pipeline-cps-method-mismatches/"
[07:39:47] <elukey>	 it wasn't there yesterday so I am wondering if anything changed on your side
[08:50:02] <hashar>	 elukey: that is apparently filed as https://phabricator.wikimedia.org/T331497
[08:50:50] <elukey>	 hashar: thanks!
[08:55:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10Shizhao) I created a Chinese test environment in patchdemo, which can be used to test PageTriag...
[08:57:45] <hashar>	 elukey: I am going to revert that pipelinelib change made by Lucas_WMDE and we merged recently :)
[08:57:55] <hashar>	 the code looked fine though and we DO have test coverage
[08:58:02] <hashar>	 but somehow all stuff is not necessarily caught
[08:59:39] <elukey>	 ack!
[09:04:47] <wikibugs>	 (03PS1) 10Hashar: Revert "Allow more BuildKit frontend image names" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/895673 (https://phabricator.wikimedia.org/T331497)
[09:04:58] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Revert "Allow more BuildKit frontend image names" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/895673 (https://phabricator.wikimedia.org/T331497) (owner: 10Hashar)
[09:05:06] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Revert "Allow more BuildKit frontend image names" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/895673 (https://phabricator.wikimedia.org/T331497) (owner: 10Hashar)
[09:06:06] <wikibugs>	 (03CR) 10Hashar: "I had to revert it because of T331497.  Despite the code looking good and test covering the feature, some magic happens in Jenkins which m" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE))
[09:06:38] <hashar>	 fun
[09:08:08] <wikibugs>	 (03CR) 10Hashar: [V: 03+2 C: 03+2] "I gotta force merge this cause the build uses the currently deployed pipelinelib to execute the pipeline and it is currently broken.  Tent" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/895673 (https://phabricator.wikimedia.org/T331497) (owner: 10Hashar)
[09:09:28] <hashar>	 elukey: I did a recheck on https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/895132
[09:09:42] <hashar>	 funnilly integration/pipelinelib (which is the code holding the repo) ended up being broken
[09:09:57] <hashar>	 cause when we +2 a change to it the jobs run with whatever is currently deployed rather than the proposed patchset
[09:10:00] <hashar>	 which leads to breakage
[09:10:03] <hashar>	 but that is another topic
[09:10:08] <hashar>	 hopefully the revert fixes it
[09:12:22] <hashar>	 I think that is fixed
[09:18:18] <wikibugs>	 (03PS1) 10Santhosh: Define test and publish pipeline for mediawiki/services/machinetranslation [integration/config] - 10https://gerrit.wikimedia.org/r/895698 (https://phabricator.wikimedia.org/T331505)
[09:19:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10SRE, 10Patch-For-Review: git: detected dubious ownership in repository at '/srv/mediawiki-staging' - https://phabricator.wikimedia.org/T325128 (10hashar) Part of the fix for the deployment servers and `scap` is https://gerrit.wikimedia.or...
[09:25:56] <elukey>	 hashar: yep all good thanks!
[09:28:34] <hashar>	 elukey: thanks for the IRC ping or we would surely have missed it :]
[10:06:22] <wikibugs>	 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Add safeguard flag to gitlab-restore.sh script - https://phabricator.wikimedia.org/T331295 (10Jelto) 05Open→03Resolved The restore script has a safeguard implemented on production (see change above):  ` gitlab2002:~$ sudo /srv/gitlab-bac...
[10:29:38] <wikibugs>	 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10Clement_Goubert) >>! In T331378#8673281, @hnowlan wrote: > There are UID differences between the two hosts:  > ` > hnowlan@deploy1002:~$ id trebuche...
[10:31:52] <wikibugs>	 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10taavi) See also: {T163667}
[10:35:05] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Excellent!" [integration/config] - 10https://gerrit.wikimedia.org/r/895698 (https://phabricator.wikimedia.org/T331505) (owner: 10Santhosh)
[10:36:20] <wikibugs>	 (03Merged) 10jenkins-bot: Define test and publish pipeline for mediawiki/services/machinetranslation [integration/config] - 10https://gerrit.wikimedia.org/r/895698 (https://phabricator.wikimedia.org/T331505) (owner: 10Santhosh)
[10:36:57] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/895698 | T331505
[10:37:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:37:02] <stashbot>	 T331505: Self hosted machine translation service - https://phabricator.wikimedia.org/T331505
[11:59:13] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10Stang) [[ https://zh.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed | Special:NewPagesFee...
[12:02:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10Stang) >>! In T323378#8671624, @kostajh wrote: > I might be missing something, but I believe `w...
[12:09:28] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10Stang)
[12:09:39] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Chinese-Sites: Incorrect create/edit time displayed in the curation toolbar - https://phabricator.wikimedia.org/T323648 (10Stang) 05Open→03Resolved a:03MPGuy2824 I could confirm this bug is fixed. Thanks!
[12:54:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests: Console error on all pages in beta-meta: ConfigException: $wgKartographerNearby requires GeoData and CirrusSearch extensions - https://phabricator.wikimedia.org/T331528 (10Daimona)
[13:00:50] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests, 10Maps (Kartographer): Console error on all pages in beta-meta: ConfigException: $wgKartographerNearby requires GeoData and CirrusSearch extensions - https://phabricator.wikimedia.org/T331528 (10Daimona)
[13:09:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10Shizhao) The workflows of zhwiki:    # “Add tags” is the same as enwiki   # speedydelete is the...
[13:38:36] <wikibugs>	 10Phabricator: Mark Maniphestbot Phabricator account as bot in database - https://phabricator.wikimedia.org/T331529 (10Aklapper) p:05Triage→03Low
[14:28:07] <wikibugs>	 (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Renameuser] Move to non-production block [integration/config] - 10https://gerrit.wikimedia.org/r/895797
[14:46:19] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Run docker-gc on deploy servers - https://phabricator.wikimedia.org/T329678 (10jnuche) docker-gc needs a state file created by the [[ https://gitlab.wikimedia.org/dancy/docker-gc/-/blob/master/docker-resource-access-monitor.py | docker-resource-access-monit...
[14:46:27] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Run docker-gc on deploy servers - https://phabricator.wikimedia.org/T329678 (10jnuche) 05In progress→03Resolved
[16:47:19] <hashar>	 jnuche: I did some review on https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/20 ;)
[16:47:23] <hashar>	 not much concern really
[16:47:33] <hashar>	 thanks for the VM!!!!!!!!!!!!!!!!! \o/
[16:48:24] <jnuche>	 hashar: thanks!
[16:56:00] <wikibugs>	 (03PS5) 10JHathaway: helm-linter: add semver-cli [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554)
[16:56:48] <jhathaway>	 hashar: any chance you could merge this in, the related patch was approved, https://gerrit.wikimedia.org/r/c/integration/config/+/893074
[17:13:27] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Move Helm chart installation out of .gitlab-ci.yml to Terraform - https://phabricator.wikimedia.org/T331549 (10demon)
[17:13:29] <wikibugs>	 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: gitlab-phabricator may be missing posts for merge request changes - https://phabricator.wikimedia.org/T329793 (10brennen) 05Open→03Resolved Haven't noticed anything recently. Will reopen if i...
[17:13:39] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Move Helm chart installation out of .gitlab-ci.yml to Terraform - https://phabricator.wikimedia.org/T331549 (10demon) 05Open→03Resolved
[17:16:38] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Reggie raises frequent "sqlite3.OperationalError: database is locked" under high load - https://phabricator.wikimedia.org/T330239 (10thcipriani) a:03dancy
[17:16:48] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Reggie raises frequent "sqlite3.OperationalError: database is locked" under high load - https://phabricator.wikimedia.org/T330239 (10thcipriani) p:05Triage→03Medium
[17:18:45] <jhathaway>	 or is anyone else around from releng, who could approve, https://gerrit.wikimedia.org/r/893074?
[17:21:16] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Add DigitalOcean resource monitoring for cloud runner nodes - https://phabricator.wikimedia.org/T308615 (10thcipriani) 05Open→03Resolved a:03thcipriani We collect metrics now.  What we have: * Digital o...
[17:21:18] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10thcipriani)
[17:23:42] <jhathaway>	 brennen: any chance you could merge it?
[17:25:21] <brennen>	 jhathaway: i don't have any context.  stuck in meetings at the moment, i can have a look after.
[17:25:31] <jhathaway>	 brennen: thanks
[17:25:50] <brennen>	 if it needs real review i don't know anything about helm-linter; if it's safe to publish i can probably do the mechanics of that.
[17:28:10] <jhathaway>	 I don't think it needs a real review, it only affects one container, and just adds an additional binary, that will be used by tests in a related patch
[17:30:31] <brennen>	 cool cool.  on my list for after this meeting.
[17:30:46] <jhathaway>	 thanks
[17:38:40] <wikibugs>	 (03PS1) 10Sbisson: Add Echo as a dependency to Wikistories [integration/config] - 10https://gerrit.wikimedia.org/r/895839
[17:41:13] <wikibugs>	 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed: Add --pause-after-testserver-sync option to deploy_security.py - https://phabricator.wikimedia.org/T328667 (10sbassett) @dancy @kostajh et al - Can we resolve this for now?  The issue was addressed but I guess technically the problem wasn't so...
[18:16:34] <wikibugs>	 (03PS1) 10Dduvall: Revert "Revert "Allow more BuildKit frontend image names"" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/895775
[18:54:37] <wikibugs>	 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed: Add --pause-after-testserver-sync option to deploy_security.py - https://phabricator.wikimedia.org/T328667 (10Tgr) It was reverted so either the task should be declined or someone needs to figure out the correct way to re-add it, I think.  (At...
[19:03:26] <wikibugs>	 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10thcipriani)
[19:03:31] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T330204 (10thcipriani)
[19:04:10] <wikibugs>	 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10thcipriani) (removing this as a train blocker—fixed enough to no longer blocking deploys, but there are lingering followups we may want to address h...
[19:16:20] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Run docker-gc on deploy servers - https://phabricator.wikimedia.org/T329678 (10dancy) I'm not understanding how this ticket has been resolved.  I don't see any reference to puppet changes to ensure that docker-gc is installed/configured on the deploy server...
[19:40:52] <wikibugs>	 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Run docker-gc on deploy servers - https://phabricator.wikimedia.org/T329678 (10jnuche) 05Resolved→03Open Sorry, I misunderstood the intention of the task. I was actually going to propose a follow-up task to automate this.  Reopened and I will add the Pu...
[19:54:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10kostajh) a:03kostajh
[19:55:17] <jhathaway>	 brennen: did you get a chance to take a look at https://gerrit.wikimedia.org/r/c/integration/config/+/893074/
[19:55:54] <brennen>	 jhathaway: yeah, just getting to that
[19:56:02] <jhathaway>	 great, thanks!
[19:56:20] <brennen>	 i just need to remember the procedure for this - going to merge and publish i guess.
[19:57:14] <wikibugs>	 (03CR) 10Brennen Bearnes: [C: 03+2] "Merging to publish per discussion with jhathaway in #wikimedia-releng." [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554) (owner: 10JHathaway)
[19:58:23] <wikibugs>	 (03Merged) 10jenkins-bot: helm-linter: add semver-cli [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554) (owner: 10JHathaway)
[20:11:21] <wikibugs>	 (03PS1) 10Krinkle: doc: Add 'Test coverage' link for MW core and a few others [integration/docroot] - 10https://gerrit.wikimedia.org/r/895857
[20:12:02] <jhathaway>	 brennen: let me know when the build is done, so I can pull down 0.4.4
[20:12:50] <brennen>	 !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/893074
[20:12:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:13:12] <brennen>	 will do.
[20:15:09] <hashar>	 I am going to upgrade the release Jenkins for a security  update  and will follow with the CI Jenkins when there are no deployment ongoing
[20:19:35] <hashar>	 of course scap goes against me :)
[20:19:44] <hashar>	 20:19:20 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'releng/jenkins-deploy', '--force', '-g', 'default', 'fetch', '--refresh-config'] (ran as deploy-jenkins@releases2002.codfw.wmnet) returned [255]: sign_and_send_pubkey: signing failed: agent refused operation
[20:19:44] <hashar>	 deploy-jenkins@releases2002.codfw.wmnet: Permission denied (publickey,keyboard-interactive).
[20:21:19] <brennen>	 jhathaway's docker image is hopefully just about built and i'll be out of the way.
[20:21:31] <jhathaway>	 much appreciated!
[20:24:01] <brennen>	 jhathaway: {{done}}
[20:24:10] <jhathaway>	 sweet!
[20:37:05] <mutante>	 hashar: maybe because "jenkins-deploy" user does not exist on releases*  and/or is not in "trusted groups". but then.. it also does not exist on contint* so .. that is still strange
[20:37:35] <mutante>	 oh, deploy-jenkins is the username
[20:38:05] <wikibugs>	 10Scap, 10Keyholder, 10serviceops, 10Datacenter-Switchover: scap can not ssh with keyholder on deploy2002 - https://phabricator.wikimedia.org/T331568 (10hashar)
[20:38:08] <mutante>	 deploy-jenkins user exists, but maybe it's not "trusted_group"
[20:38:13] <hashar>	 nop
[20:38:20] <hashar>	 it is working on deploy1002
[20:38:33] <hashar>	 I filed my investigation as https://phabricator.wikimedia.org/T331568
[20:38:34] <mutante>	 ah, so the difference is on deployment server side, gotcha
[20:38:40] <mutante>	 ack
[20:39:01] <hashar>	 my theory is that releng/jenkins-deploy has been added recently
[20:39:06] <hashar>	 the keyholder got armed on both hosts
[20:39:24] <hashar>	 but there is an extra yaml file that does the permission and it requires the proxy to be restarted
[20:39:43] <hashar>	 so I guess the keyholder-proxy service needs a restart on deploy2002 to take in accunt the latest permissions
[20:39:51] <hashar>	 :]
[20:40:09] <hashar>	 can be tested with `SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet`
[20:40:15] <hashar>	 I have to go AFK a bit
[20:40:19] <mutante>	 hashar: restart done :p
[20:40:46] <mutante>	 hashar: fixed. SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -i /etc/keyholder.d/deploy_jenkins -l deploy-jenkins releases1002.eqiad.wmnet  works now
[20:41:47] <wikibugs>	 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed: Add --pause-after-testserver-sync option to deploy_security.py - https://phabricator.wikimedia.org/T328667 (10dancy) >>! In T328667#8677203, @sbassett wrote: > @dancy @kostajh et al - Can we resolve this for now?  The issue was addressed but I...
[20:44:16] <wikibugs>	 10Release-Engineering-Team, 10SecTeam-Processed: deploy_security.py hangs after scap - https://phabricator.wikimedia.org/T329602 (10sbassett) 05Open→03Resolved p:05Triage→03Medium a:03sbassett
[20:47:41] <wikibugs>	 10Scap, 10Keyholder, 10serviceops, 10Datacenter-Switchover: scap can not ssh with keyholder on deploy2002 - https://phabricator.wikimedia.org/T331568 (10Dzahn) 05Open→03Resolved a:03Dzahn should be fixed. By restarting the proxy as you suggested. Test below works:  ` [deploy2002:~] $ SSH_AUTH_SOCK=/r...
[20:51:11] <hashar>	 mutante: cool! I will try with scap in a few.  I guedd the DC switch over doc must document the proy has to be restarted
[20:51:26] <wikibugs>	 10Scap, 10Keyholder, 10serviceops, 10serviceops-collab, 10Datacenter-Switchover: scap can not ssh with keyholder on deploy2002 - https://phabricator.wikimedia.org/T331568 (10Dzahn)
[20:51:47] <hashar>	 until keyholder is made smarter and get to reload its conf whenever a file is touched. We had a parch for it i think
[20:53:43] <mutante>	 ok, cool, yea
[20:55:14] <dancy>	 yak shaving!
[20:59:25] <hashar>	 that is what I was referring to in our team meeting earlier :)
[21:01:12] <mutante>	 everytime deployment server is switched there is a bunch of those
[21:01:16] <mutante>	 afair
[21:02:32] <wikibugs>	 10Scap, 10Keyholder, 10serviceops, 10serviceops-collab, 10Datacenter-Switchover: scap can not ssh with keyholder on deploy2002 - https://phabricator.wikimedia.org/T331568 (10hashar) 05Resolved→03Open I have confirmed it works. I am reopening so that the #datacenter-switchover documentation gets updat...
[21:02:45] <mutante>	 likely though we'll end up saying that something is going to change anyways soon, so it's not worth it.. but then we do it again :)
[21:02:47] <hashar>	 mutante: I have reopened it for documentation purpose, I will chat with Clément  about it
[21:02:54] <mutante>	 ok
[21:03:17] <hashar>	 I am 100% sure we had a patch to have the keyholder service to reload the config when receiving a SIGHUP
[21:03:25] <hashar>	 either it never got merged or never got released/deployed
[21:03:28] <hashar>	 OR
[21:03:35] <hashar>	 we forgot to make Puppet to trigger a reload
[21:03:45] <hashar>	 anyway, gotta do the Jenkins upgrade now :]
[21:04:38] <mutante>	 or it's that keyholder-agent is restarted but keyholder-proxy is not
[21:04:47] <mutante>	 both are separate systemd service
[21:20:56] <hashar>	 mutante: I forgot, thank you for the quick keyholder-proxy restart :-]
[21:24:00] <mutante>	 hashar: no problem, yw
[21:25:23] <wikibugs>	 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Onboarding 🚀), 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10thcipriani) >>! In T285354#7171268, @Krinkle wrote: > Consider enforcing the privacy policy we c...
[21:28:27] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Upstream: GitLab truncates commit messages over 1k of text - https://phabricator.wikimedia.org/T330790 (10brennen)
[21:28:37] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team, 10Upstream: Gitlab message after push point to a 404 merge request URL when not logged in - https://phabricator.wikimedia.org/T324262 (10brennen)
[21:28:44] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Accessibility, 10Upstream: Gitlab announce banner / broadcast message are unreadable due to dark background - https://phabricator.wikimedia.org/T330496 (10brennen)
[21:28:54] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Upstream: Forking a public project without selecting a visibility level fails - https://phabricator.wikimedia.org/T323361 (10brennen)
[21:32:32] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Radar), 10Upstream, 10User-brennen: Look into whether GitLab time tracking can be disabled - https://phabricator.wikimedia.org/T264230 (10thcipriani) a:05brennen→03None
[21:34:24] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10MediaWiki-extensions-Gadgets, 10Security-Team, 10Security: Allow Javascript files from Wikimedia GitLab to be loaded as scripts in Wikimedia wikis - https://phabricator.wikimedia.org/T321458 (10thcipriani)
[21:34:32] <wikibugs>	 (03CR) 10Hashar: "+1 :) I wasn't sure whether `semver-cli` would still be kept in the deployment-charts proposed patchset.   Then it would have been easy to" [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554) (owner: 10JHathaway)
[21:34:36] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Priority Backlog 📥), 10SecTeam-Processed, 10Security, 10Upstream: Enabling CORS for raw file URLs - https://phabricator.wikimedia.org/T305700 (10thcipriani)
[21:34:51] <wikibugs>	 (03CR) 10Hashar: helm-linter: add semver-cli (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/893074 (https://phabricator.wikimedia.org/T320554) (owner: 10JHathaway)
[21:40:28] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10Security-Team, 10User-AKlapper, 10user-sbassett: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper) a:03Bmueller
[21:45:54] <jhathaway>	 anyone know how to clear the container cache on ci, so it grabs the latest image?
[21:58:19] <wikibugs>	 (03PS1) 10JHathaway: helm-linter: bump version [integration/config] - 10https://gerrit.wikimedia.org/r/895868 (https://phabricator.wikimedia.org/T320554)
[21:58:46] <hashar>	 jhathaway: we dont ;)
[21:58:59] <hashar>	 it is version pinned indeed
[21:59:07] <jhathaway>	 :)
[21:59:23] <jhathaway>	 if you could merge that it hashar, that would be great
[21:59:45] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "./jjb-update helm-lint" [integration/config] - 10https://gerrit.wikimedia.org/r/895868 (https://phabricator.wikimedia.org/T320554) (owner: 10JHathaway)
[21:59:49] <hashar>	 deployed
[22:00:00] <jhathaway>	 woohoo
[22:00:17] <hashar>	 jhathaway: I guess we should open a thread somewhere to organize some training to give you all more freedom on that
[22:00:31] <hashar>	 it originally started as a repo mostly managed by me and a few folks
[22:00:41] <hashar>	 and it was very very easy to break everything entirely (and still is)
[22:00:59] <hashar>	 but nowadays in most case people tweak an image then just have to do the version bump as you did above so there are less risk I guess
[22:01:13] <wikibugs>	 (03Merged) 10jenkins-bot: helm-linter: bump version [integration/config] - 10https://gerrit.wikimedia.org/r/895868 (https://phabricator.wikimedia.org/T320554) (owner: 10JHathaway)
[22:01:46] <hashar>	 I am still around for a few if you wanna try the update job
[22:01:58] <jhathaway>	 tough problem to solve
[22:02:56] <hashar>	 possibly :]
[22:03:30] <jhathaway>	 I just reran the job, and it picked up the correct version, thanks hashar 
[22:03:48] <hashar>	 the Jenkins/Zuul started as an intiative to test MediaWiki and was centrally managed and under just a very few people
[22:04:15] <hashar>	 but it has grown organically like a bacteria and end sup being on the critical path of a lot of things :D
[22:04:19] <hashar>	 great
[22:04:36] <hashar>	 jhathaway: if you don't need that semver thing later on , remember to drop it from the image eventually
[22:04:43] <jhathaway>	 hashar: will do
[22:04:53] <hashar>	 and if you ever need to update that cli tool, the same config dance will need to be done
[22:05:03] <hashar>	 a dockerfile.template tweak / changelog update
[22:05:07] <jhathaway>	 makes sense, I'll try to remember :)
[22:05:15] <hashar>	 and bumping the image in the Jenkins  job
[22:05:38] <hashar>	 we don't use :latest cause that would require all jobs to always pull the image which migth well criplle the registry
[22:05:44] <hashar>	 and we want to control when the jobs are upgrade
[22:05:56] <hashar>	 so we can create a new image, try out locally or on some job, then generalize to everything
[22:05:59] <hashar>	 something like that :]
[22:06:02] <jhathaway>	 makes sense, I was just confused because the in repo rake task used latest
[22:06:20] <hashar>	 ah yeah
[22:06:58] <hashar>	 which lets you verify locally everything works fine before promoting the image to CI
[22:07:11] <hashar>	 maybe one day the version will be defined via an etcd store or something who knows
[22:07:18] <hashar>	 happy hacking!
[22:07:24] <jhathaway>	 thanks!
[22:49:23] <hashar>	 I have updated the CI Jenkins from 2.375.3 to 2.375.4
[23:29:17] <wikibugs>	 10Project-Admins, 10Data-Persistence (work done), 10Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-issue - https://phabricator.wikimedia.org/T305639 (10Aklapper) >>! In T305639#8134155, @Ladsgroup wrote: > Let's look at this once Manuel is back  I gues...