[00:36:09] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10brennen) Cloud runner version of above change: https://gitlab.wikimedia.org/repos... [00:42:21] !log on deployment-deploy03 removed helm2, as was done in production [00:42:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:33:25] 10Project-Admins: Create project tag for Security API Service - https://phabricator.wikimedia.org/T305728 (10sbassett) p:05Triage→03Lowest [05:22:25] Project beta-update-databases-eqiad build #59272: 04FAILURE in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59272/ [06:22:24] Project beta-update-databases-eqiad build #59273: 04STILL FAILING in 2 min 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59273/ [06:37:35] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: InvalidArgumentException: Unknown cluster 'extension2' - https://phabricator.wikimedia.org/T310569 (10dom_walden) [06:42:58] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: InvalidArgumentException: Unknown cluster 'extension2' - https://phabricator.wikimedia.org/T310569 (10RhinosF1) p:05Triage→03Unbreak! This is likely the deploy by @TimStarling - issue was already mentioned on IRC & https://gerrit.wikimedia.org/r/c... [06:51:00] (03PS1) 10Jeena Huneidi: scap.cfg: Remove mwdebug_hosts [tools/train-dev] - 10https://gerrit.wikimedia.org/r/805312 (https://phabricator.wikimedia.org/T308476) [06:53:24] (03PS18) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) [06:54:10] (03PS19) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) [06:57:29] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [07:17:29] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10Patch-For-Review: InvalidArgumentException: Unknown cluster 'extension2' - https://phabricator.wikimedia.org/T310569 (10tstarling) 05Open→03Resolved a:03tstarling Looks like that fixed it. I also created the table. [07:20:18] thcipriani: hashar: hello, may I ask what is T309375 blocked/stalled on? [07:25:23] Yippee, build fixed! [07:25:24] Project beta-update-databases-eqiad build #59274: 09FIXED in 5 min 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59274/ [09:03:32] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10hashar) Indeed, I am sure @jnuche and I can address this one :] [09:05:17] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10hashar) [09:06:24] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10jnuche) Yeah, @hashar and I will pair on this [09:18:13] 10Project-Admins: Create project tag for Gitlab Application Security Pipeline - https://phabricator.wikimedia.org/T310563 (10Aklapper) [10:32:45] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10JAllemandou) [10:35:23] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10JAllemandou) [10:39:22] Hi, this patch https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/790396 is merged for quite some time now but I dont see the deployment window under https://wikitech.wikimedia.org/wiki/Deployments. Is there something else we need to do for the new deployment window ? [10:48:55] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10hashar) a:03hashar That is more or less a recurring issue as I understand it. The WMCS instances are in the `gitlab-runners` WMCS project, there is no monitoring for them. Diamond has been phased ou... [10:59:07] nemo-yiannis: Strange. I'd ask thcipriani. I think he's the one running the calendar script :) [10:59:18] the calendar patch looks good to me [10:59:18] Thanks hauskatze [11:38:37] https://make.wordpress.org/core/2022/06/14/upcoming-disallow-assignments-in-conditions-and-remove-the-yoda-condition-requirement-for-php/ [11:39:07] Nice to see how they use phpcs on a similarly large and open php based eco system [11:46:13] 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10hashar) 05Open→03Resolved [12:27:57] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10PleaseStand) After the 1.39.0-wmf.16 branch cut happened, @tstarling deployed a change to the AbuseFilter extension (see T212129)... [12:33:40] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10tstarling) Yes, it is necessary. Reverting it would overload the x2 cluster. [12:36:04] (03PS1) 10Hashar: scap prep: set ssh push url on all repositories [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) [12:53:38] (03CR) 10Hashar: "I got it configured in my ~/.gitconfig since https://gerrit.wikimedia.org/r/c/operations/puppet/+/510119/2/modules/admin/files/home/hashar" [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [13:34:01] (03PS2) 10Hashar: scap prep: set ssh push url on all repositories [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) [13:34:04] Project beta-update-databases-eqiad build #59280: 15ABORTED in 19 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/59280/ [13:35:16] !log clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer [13:35:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:37:29] 10Project-Admins: Create project tag for Gitlab Application Security Pipeline - https://phabricator.wikimedia.org/T310563 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #Gitlab-Application-Security-Pipeline has been created: https://phabricator.wikimedia.org/project/view/5984/ (In ca... [13:58:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Move doc.wikimedia.org storage to a distributed storage - https://phabricator.wikimedia.org/T310617 (10hashar) [14:16:37] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10Jdforrester-WMF) [14:17:28] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10Jdforrester-WMF) >>! In T308069#8002312, @tstarling wrote: > Yes, it is necessary. Reverting it would overload the x2 cluster. M... [14:24:34] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10hashar) The idea is to have scap to push to Gerrit with the ssh user set to `trainbranchbot`. That can be done by running `git p... [15:10:21] (03CR) 10Ahmon Dancy: [C: 04-1] scap prep: set ssh push url on all repositories (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [15:16:49] (03CR) 10Ahmon Dancy: [C: 03+2] scap.cfg: Remove mwdebug_hosts [tools/train-dev] - 10https://gerrit.wikimedia.org/r/805312 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [15:17:21] (03Merged) 10jenkins-bot: scap.cfg: Remove mwdebug_hosts [tools/train-dev] - 10https://gerrit.wikimedia.org/r/805312 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [15:21:30] (03PS1) 10Jaime Nuche: install-world: remove concurrency lock [tools/scap] - 10https://gerrit.wikimedia.org/r/805406 (https://phabricator.wikimedia.org/T303559) [15:40:41] (03CR) 10Ahmon Dancy: [C: 03+2] install-world: remove concurrency lock [tools/scap] - 10https://gerrit.wikimedia.org/r/805406 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche) [15:43:59] (03CR) 10Ahmon Dancy: "I will test" [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [15:48:13] (03Merged) 10jenkins-bot: install-world: remove concurrency lock [tools/scap] - 10https://gerrit.wikimedia.org/r/805406 (https://phabricator.wikimedia.org/T303559) (owner: 10Jaime Nuche) [15:50:52] (03PS1) 10Jaime Nuche: Release 4.9.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805413 [15:51:21] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [15:55:23] (03CR) 10Jaime Nuche: [C: 03+2] Release 4.9.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805413 (owner: 10Jaime Nuche) [15:58:20] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10TheresNoTime) Ref the [[ https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/805406 |... [15:59:36] (03Merged) 10jenkins-bot: Release 4.9.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/805413 (owner: 10Jaime Nuche) [16:00:23] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) > Marked as a blocker and cherry-picked https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/805361; sorr... [16:05:15] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10dduvall) 05Open→03In progress p:05Triage→03Medium a:03dduvall [16:05:18] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Design m8s deployment workflows and tooling - https://phabricator.wikimedia.org/T279322 (10dduvall) [16:12:55] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10dduvall) [16:15:01] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) @TheresNoTime Thanks for the report! [16:20:15] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10jnuche) hi @TheresNoTime, thank you for your comment! The scap operation running in beta... [16:20:27] 10Phabricator: Should not require an NDA privilege to see "In Phabricator I see a field called “Other assignee” field - https://phabricator.wikimedia.org/T310638 (10Jdlrobson) [16:37:55] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) [16:52:35] stupid question time: anything i need to know about current state of deployment tooling that isn't represented in current state of train docs? [16:52:44] ^ cc: dancy, jnuche, jeena [16:53:37] Nope. You should be able to use `scap stage-train`. I recommend passing in the `--yes` option so we can exercise what we hope that the train auto-sync will do. [16:53:57] s/--yes/-y/ [16:54:57] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: scap backport --revert command - https://phabricator.wikimedia.org/T287046 (10jeena) a:03jeena [17:03:53] 10Release-Engineering-Team, 10Platform Engineering, 10Similar Editors, 10Anti-Harassment (AHaT Sprint 10: The Mokorotlo): Configure SimilarEditors in production with Similarusers credentials - https://phabricator.wikimedia.org/T308670 (10ARamirez_WMF) [17:04:12] dancy: ack, will do. [17:18:03] !log starting 1.39.0-wmf.16 (T308069) transcript in deploy1002:~brennen/1.39.0-wmf.16.log [17:18:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:18:07] T308069: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 [17:24:13] hrm: [17:24:16] 17:20:34 Invalid JSON received from https://train-blockers.toolforge.org/api.php: [17:24:18] Expecting value: line 1 column 1 (char 0) [17:24:28] bogus! [17:24:49] That's from `scap stage-train` ? [17:25:11] yeah [17:25:18] ok.. taking a look [17:25:20] scrollback in ~brennen/1.39.0-wmf.16.log if you're curious [17:26:09] The tail of that output hasn't been flushed yet [17:27:45] Try re-running please. [17:28:37] it's still running [17:29:04] oh interesting. It continued past the complaint? [17:29:26] ah, I see from the code that it just warns and returns a blank string.. hmm [17:29:34] yeah - i'm wondering if it just doesn't have the task? [17:30:09] `env https_proxy=http://webproxy:8080 curl https://train-blockers.toolforge.org/api.php | jq .current.task_id` works fine [17:30:50] yeah, sure enough [17:31:03] I'll update the warning so that it shows what it got from the request. [17:31:23] cool, thx for looking [17:32:06] (03PS1) 10Ahmon Dancy: scap prep: Print response from https://train-blockers.toolforge.org/api.php if not parseable as JSON [tools/scap] - 10https://gerrit.wikimedia.org/r/805453 [17:33:22] (03CR) 10Brennen Bearnes: [C: 03+2] scap prep: Print response from https://train-blockers.toolforge.org/api.php if not parseable as JSON [tools/scap] - 10https://gerrit.wikimedia.org/r/805453 (owner: 10Ahmon Dancy) [17:36:56] (03Merged) 10jenkins-bot: scap prep: Print response from https://train-blockers.toolforge.org/api.php if not parseable as JSON [tools/scap] - 10https://gerrit.wikimedia.org/r/805453 (owner: 10Ahmon Dancy) [17:42:34] 10Release-Engineering-Team, 10Web Team Visual Regression Framework, 10Readers-Web-Backlog (Needs Prioritization (Tech)): Make visual regression tests run in CI (non-blocking) for the Vector repo - https://phabricator.wikimedia.org/T308194 (10Jdlrobson) [17:44:31] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [17:46:37] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [17:53:40] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [17:56:17] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [17:56:27] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [17:57:17] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [18:49:59] (03PS1) 10Daimona Eaytoy: zuul: [mediawiki/extensions/WikimediaMessages] Add CampaignEvents as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/805458 [18:51:28] 10Phabricator: Should not require an NDA privilege to see "In Phabricator I see a field called “Other assignee” field - https://phabricator.wikimedia.org/T310638 (10Raphoraph) The title seems strange, should it rather be “Should not require an NDA privilege to see in Phabricator the “Other assignee” field” ? [18:51:57] (03CR) 10CI reject: [V: 04-1] zuul: [mediawiki/extensions/WikimediaMessages] Add CampaignEvents as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/805458 (owner: 10Daimona Eaytoy) [18:54:04] (03PS2) 10Daimona Eaytoy: zuul: Add CampaignEvents as phan dependency for WikimediaMessages [integration/config] - 10https://gerrit.wikimedia.org/r/805458 [19:00:56] 10Phabricator: Should not require an NDA privilege to see in Phabricator the “Other assignee” field - https://phabricator.wikimedia.org/T310638 (10Jdlrobson) [19:02:19] 10Phabricator: Should not require an NDA privilege or special template to see in Phabricator the “Other assignee” field - https://phabricator.wikimedia.org/T310638 (10Jdlrobson) Fixed. Also interestingly, I can't see the other assignee on field on this task (but can on T300612). It would be good to consistently... [19:02:26] 10Phabricator: Should not require an NDA privilege or special template to see in Phabricator the “Other assignee” field - https://phabricator.wikimedia.org/T310638 (10Jdlrobson) [19:05:58] 10Phabricator: Rename my phabricator account to "Baggaet". - https://phabricator.wikimedia.org/T310651 (10Pppery) [19:12:25] (03PS1) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) [19:12:30] (03PS20) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) [19:13:12] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [19:14:08] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [19:15:05] (03CR) 10Hashar: [C: 04-2] Support a different user for Gerrit ssh interactions (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [20:29:03] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Infrastructure-Foundations, 10SRE-Access-Requests: Allow deployers to sudo -u mwpresync - https://phabricator.wikimedia.org/T310654 (10dancy) [20:29:34] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests: Allow deployers to sudo -u mwpresync - https://phabricator.wikimedia.org/T310654 (10dancy) [20:29:38] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) [20:33:58] (03PS1) 10Ahmon Dancy: test-scap-backport: Start ssh-agent earlier [tools/train-dev] - 10https://gerrit.wikimedia.org/r/805473 [20:39:43] 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10thcipriani) [20:43:50] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [20:46:14] (03CR) 10Hashar: scap prep: set ssh push url on all repositories (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [20:47:59] (03CR) 10Ahmon Dancy: [C: 04-1] scap prep: set ssh push url on all repositories (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [20:48:05] lovely scap `cli.Application` [21:00:28] 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10thcipriani) In case it's helpful here, here's a list of everything that went live during that spike: https://www.mediawiki.org/wiki/MediaWiki_1... [21:11:41] (03PS21) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) [21:11:56] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:12:15] (03CR) 10Jeena Huneidi: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:18:11] (03CR) 10Ahmon Dancy: [C: 03+1] "Code looks good to me. Starting a test run now." [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:21:19] 10Deployments, 10Release-Engineering-Team (Doing), 10SRE, 10bacula, 10Parsoid (Tracking): Accidental removal of some files under /srv/deployment on deploy1002 - https://phabricator.wikimedia.org/T307349 (10ssastry) [21:27:37] (03CR) 10Ahmon Dancy: [C: 03+1] scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:30:34] !log clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date [21:30:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:31:12] "monitored for", great English :') [21:34:32] (03PS22) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) [21:38:38] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:41:57] (03PS23) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) [21:42:15] 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10Jdforrester-WMF) Is this just the UBN-ness of the back-ported fixes to AbuseFilter for {T212129}? [21:43:17] TheresNoTime: we English are worst at speaking English [21:43:36] It's also a super weird language that makes no sense [21:46:25] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:50:44] (03CR) 10Ahmon Dancy: [C: 04-1] scap prep: set ssh push url on all repositories (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805373 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [21:50:49] (03CR) 10CI reject: [V: 04-1] scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:53:39] (03CR) 10Ahmon Dancy: [V: 03+2] scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [21:53:56] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] "Trying again" [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [22:02:29] (03Merged) 10jenkins-bot: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [22:03:54] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] scap backport: deploy to mwdebug (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [22:34:48] (03CR) 10Jeena Huneidi: [C: 03+2] "tested, LGTM" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/805473 (owner: 10Ahmon Dancy) [22:35:14] (03Merged) 10jenkins-bot: test-scap-backport: Start ssh-agent earlier [tools/train-dev] - 10https://gerrit.wikimedia.org/r/805473 (owner: 10Ahmon Dancy) [23:26:23] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10dduvall) Putting this up for grabs since I'll be out on vacation for the remainder of the week. Upon investigation, implementing `scap... [23:27:04] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10dduvall) 05In progress→03Open a:05dduvall→03None [23:27:07] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Design m8s deployment workflows and tooling - https://phabricator.wikimedia.org/T279322 (10dduvall) [23:34:36] (03PS1) 10Dduvall: history: Mark latest history entry as synced following any sync command [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) [23:35:10] you got buildkit on gitlab-runner1001 now! [23:35:29] found a magic fix for the iptables issue.. it's ... restarting docker... [23:35:37] let me deploy on other runners [23:40:46] actually..no.. after that buildkitd service can't start [23:41:04] back to reverting that because we don't want puppet to be alerting on all runners [23:41:19] but I am creating a test instance in devtools so this can be tested [23:56:46] 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10Krinkle) a:03Krinkle