[03:42:59] PROBLEM - Puppet staleness on deployment-sessionstore02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [03:43:29] PROBLEM - Puppet staleness on deployment-sessionstore01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [03:50:33] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [03:57:09] PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [05:55:43] 10Project-Admins: Create Project: Watchlist-Expiry - https://phabricator.wikimedia.org/T235686 (10Aklapper) Has anyone talked to WMDE to know if they plan to continue working on #Expiring-Watchlist-Items [again] at some point? If WMDE has no plans to work on #Expiring-Watchlist-Items, I don't see a reason why no... [06:52:17] (03CR) 10Hashar: [C: 04-1] dockerfiles: [node10-test-browser-php72-composer] Make this actually provide both PHP and Node (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/543723 (owner: 10Jforrester) [07:21:34] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:25] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 49270 bytes in 1.083 second response time [08:58:54] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Quibble, and 3 others: CI: Create a way to share a secret between MediaWiki and the testing framework. - https://phabricator.wikimedia.org/T233092 (10hashar) >>! In T233092#5564494, @dani... [09:39:45] hello! [09:40:27] I am trying to scap deploy to deployment-event05 but it returns me Permission denied (publickey). [09:41:00] yesterday was working fine, so I am wondering if something changed.. I noticed https://phabricator.wikimedia.org/T235674 from the sal [09:43:35] and SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l analytics_deploy analytics_deploy@deployment-eventlog05.deployment-prep.eqiad.wmflabs doesn't work [12:49:25] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Quibble, 10Patch-For-Review: The phpunit-coverage jobs shouldn't run npm install - https://phabricator.wikimedia.org/T225008 (10hashar) p:05Triage→03Normal [12:50:37] (03PS1) 10Jack Phoenix: Add SocialProfile and VoteNY as phan dependencies for ImageRating [integration/config] - 10https://gerrit.wikimedia.org/r/543843 [14:14:04] (03CR) 10Hashar: "Gotta give it a try more. Note that http host and ports are supposedly no more hardcoded anywhere to ensure they are consistent." [integration/quibble] - 10https://gerrit.wikimedia.org/r/516729 (https://phabricator.wikimedia.org/T225218) (owner: 10Kosta Harlan) [14:26:17] 10Project-Admins: Create Project: Watchlist-Expiry - https://phabricator.wikimedia.org/T235686 (10ifried) @JStrodt_WMDE I'm looping you into the conversation, so we can be sure that we have the correct information on WMDE Community Tech. Given that WMF Community Tech is now planning to work on Watchlist Expiry,... [14:43:26] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10BBlack) IRC says the meeting was mostly consumed by OKR discussion, it may ha... [14:45:22] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10Wikimedia Design Style Guide: Automatic pickup of Gerrit clone master doesn't happen - https://phabricator.wikimedia.org/T235677 (10crusnov) p:05Triage→03Normal [14:45:30] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Fix operations/puppet.git "rebase hell" - https://phabricator.wikimedia.org/T224033 (10BBlack) 05Open→03Resolved a:03BBlack It's switched to `Rebase-if-necess... [15:04:02] (03CR) 10Jforrester: dockerfiles: [node10-test-browser-php72-composer] Make this actually provide both PHP and Node (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/543723 (owner: 10Jforrester) [15:04:23] 10Project-Admins: Create Project: Watchlist-Expiry - https://phabricator.wikimedia.org/T235686 (10JStrodt_WMDE) >>! In T235686#5584013, @ifried wrote: > @JStrodt_WMDE I'm looping you into the conversation, so we can be sure that we have the correct information on WMDE Community Tech. Given that WMF Community Tec... [15:35:28] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: ERROR webdriver: Request failed due to Error: session not created: Chrome version must be betwe... - https://phabricator.wikimedia.org/T234610 [15:48:22] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10Wikimedia Design Style Guide: Automatic pickup of Gerrit clone master doesn't happen (due to git-lfs not installed on production misc) - https://phabricator.wikimedia.org/T235677 (10Aklapper) [15:48:49] qq - is anybody able to deploy in deployment-prep now? [15:50:11] elukey: Probably. Whats up? [15:50:44] I wrote something earlier on today, I have a problem when deploying to eventlog05 [15:50:49] and yesterday it was working [15:51:39] I tried also with SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l analytics_deploy deployment-eventlog05.deployment-prep.eqiad.wmflabs [15:51:53] IIUC the keyholder was updated yesterday [15:52:22] so I was wondering if I have a problem only on my side or if it was more widespread [16:07:33] (03PS2) 10Jforrester: dockerfiles: [node10-test-browser-php72-composer] Make this actually provide both PHP and Node [integration/config] - 10https://gerrit.wikimedia.org/r/543723 [16:07:35] (03PS4) 10Jforrester: jjb: Point OOUI experimental image at node10-test-browser-php72-composer [integration/config] - 10https://gerrit.wikimedia.org/r/543227 (https://phabricator.wikimedia.org/T235570) [16:08:40] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [node10-test-browser-php72-composer] Make this actually provide both PHP and Node [integration/config] - 10https://gerrit.wikimedia.org/r/543723 (owner: 10Jforrester) [16:08:46] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: [node10-test-browser-php72-composer] Make this actually provide both PHP and Node [integration/config] - 10https://gerrit.wikimedia.org/r/543723 (owner: 10Jforrester) [16:09:39] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/543723 (owner: 10Jforrester) [16:10:19] (03Merged) 10jenkins-bot: dockerfiles: [node10-test-browser-php72-composer] Make this actually provide both PHP and Node [integration/config] - 10https://gerrit.wikimedia.org/r/543723 (owner: 10Jforrester) [16:11:08] !log Docker: Pushing node10-test-browser-php72-composer:0.1.1 [16:11:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:22:47] James_F: o/ - sorry to ping you, I noticed https://phabricator.wikimedia.org/T235674 and today I wasn't able to deploy to deployment-eventlog05 (permission denied with ssh, also tried with SSH_AUTH_SOCK). Could it be related to the keyholder? Not sure if I am the only one with this issue [16:23:24] elukey: Yeah, it's possible. Sorry, no idea about that stuff. :-( [16:23:29] ah okok :) [16:24:31] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/543227 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:26:25] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (201910): Beta cluster doesn’t update since ca. 2019-10-15 21:00 UTC - https://phabricator.wikimedia.org/T235674 (10elukey) 05Resolved→03Open I am currently failing to deploy on deployment-deploy05: ` elukey@deployment-deploy01:/srv/deploymen... [16:26:51] re-opened the task :) [16:27:34] (03Merged) 10jenkins-bot: jjb: Point OOUI experimental image at node10-test-browser-php72-composer [integration/config] - 10https://gerrit.wikimedia.org/r/543227 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:29:03] (03PS1) 10Jforrester: layout: [OOUI] Move from node 6 to node 10 job [integration/config] - 10https://gerrit.wikimedia.org/r/543891 (https://phabricator.wikimedia.org/T235570) [16:29:06] (03PS1) 10Jforrester: jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) [16:29:08] (03PS1) 10Jforrester: jjb: Move OOUI jobs to be withthe others in oojs.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) [16:29:10] (03PS1) 10Jforrester: jjb: Move oojs-ui-docker-publish to use node10-test-browser-php72-composer too [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) [16:29:12] (03PS1) 10Jforrester: dockerfiles: Drop npm-test-oojsui, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543895 (https://phabricator.wikimedia.org/T235570) [16:29:14] (03PS1) 10Jforrester: layout: [OOUI] Drop generic-node10-rundoc-docker and …-npmaudit-docker experiments [integration/config] - 10https://gerrit.wikimedia.org/r/543896 (https://phabricator.wikimedia.org/T235570) [16:29:16] (03CR) 10Jforrester: [C: 03+2] layout: [OOUI] Move from node 6 to node 10 job [integration/config] - 10https://gerrit.wikimedia.org/r/543891 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:31:57] (03Merged) 10jenkins-bot: layout: [OOUI] Move from node 6 to node 10 job [integration/config] - 10https://gerrit.wikimedia.org/r/543891 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:33:00] https://gerrit-review.googlesource.com/c/gerrit/+/241594 that's awesome! [16:33:11] Will allow us to do Polymer.html`` [16:33:12] !log Zuul: Moving OOUI from node 6 to node 10 job T235570 [16:33:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:33:15] T235570: Move the OOUI repo to a new custom docker image for node10 and php72 - https://phabricator.wikimedia.org/T235570 [16:34:37] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:34:53] (03CR) 10Jforrester: [C: 03+2] "No-op." [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:35:06] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:35:45] (03CR) 10jerkins-bot: [V: 04-1] jjb: Move oojs-ui-docker-publish to use node10-test-browser-php72-composer too [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:35:59] (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:36:01] (03CR) 10jerkins-bot: [V: 04-1] jjb: Move OOUI jobs to be withthe others in oojs.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:36:06] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: Drop npm-test-oojsui, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543895 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:36:17] (03CR) 10jerkins-bot: [V: 04-1] layout: [OOUI] Drop generic-node10-rundoc-docker and …-npmaudit-docker experiments [integration/config] - 10https://gerrit.wikimedia.org/r/543896 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:36:31] Err. [16:36:38] ABORTED? [16:37:46] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [16:57:48] (03PS2) 10Jforrester: layout: [OOUI] Drop generic-node10-rundoc-docker and …-npmaudit-docker experiments [integration/config] - 10https://gerrit.wikimedia.org/r/543896 (https://phabricator.wikimedia.org/T235570) [16:57:50] (03PS2) 10Jforrester: jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) [16:57:52] (03PS2) 10Jforrester: jjb: Move OOUI jobs to be withthe others in oojs.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) [16:57:54] (03PS2) 10Jforrester: jjb: Move oojs-ui-docker-publish to use node10-test-browser-php72-composer too [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) [16:57:56] (03PS2) 10Jforrester: dockerfiles: Drop npm-test-oojsui, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543895 (https://phabricator.wikimedia.org/T235570) [16:57:58] (03PS1) 10Jforrester: layout: [OOUI] Drop PHP70 and PHP71 testing [integration/config] - 10https://gerrit.wikimedia.org/r/543902 (https://phabricator.wikimedia.org/T235570) [17:04:33] (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:04:50] (03CR) 10jerkins-bot: [V: 04-1] jjb: Move oojs-ui-docker-publish to use node10-test-browser-php72-composer too [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:05:13] (03CR) 10jerkins-bot: [V: 04-1] jjb: Move OOUI jobs to be withthe others in oojs.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:05:15] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: Drop npm-test-oojsui, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543895 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:06:46] (03CR) 10Jforrester: [C: 03+2] layout: [OOUI] Drop generic-node10-rundoc-docker and …-npmaudit-docker experiments [integration/config] - 10https://gerrit.wikimedia.org/r/543896 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:08:44] (03Merged) 10jenkins-bot: layout: [OOUI] Drop generic-node10-rundoc-docker and …-npmaudit-docker experiments [integration/config] - 10https://gerrit.wikimedia.org/r/543896 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:10:20] !log Zuul: OOUI] Drop generic-node10-rundoc-docker and …-npmaudit-docker experiments [17:10:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:15:55] 10Release-Engineering-Team, 10serviceops: Missing annotations for sync-wikiversions - https://phabricator.wikimedia.org/T235787 (10jijiki) [17:24:25] (03CR) 10Jforrester: [C: 03+2] layout: [OOUI] Drop PHP70 and PHP71 testing [integration/config] - 10https://gerrit.wikimedia.org/r/543902 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:26:28] (03Merged) 10jenkins-bot: layout: [OOUI] Drop PHP70 and PHP71 testing [integration/config] - 10https://gerrit.wikimedia.org/r/543902 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:30:30] (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:30:32] (03CR) 10jerkins-bot: [V: 04-1] jjb: Move OOUI jobs to be withthe others in oojs.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:30:34] (03CR) 10jerkins-bot: [V: 04-1] jjb: Move oojs-ui-docker-publish to use node10-test-browser-php72-composer too [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:47:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10JavaScript: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects - https://phabricator.wikimedia.org/T211784 (10Jdforrester-WMF) [17:47:20] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10OOUI, 10Patch-For-Review: Move the OOUI repo to a new custom docker image for node10 and php72 - https://phabricator.wikimedia.org/T235570 (10Jdforrester-WMF) 05Open→03Resol... [17:49:06] (03PS3) 10Jforrester: jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) [17:49:20] (03CR) 10Jforrester: [C: 03+2] jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [17:49:27] (03PS3) 10Jforrester: jjb: Move OOUI jobs to be withthe others in oojs.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) [17:49:37] (03PS3) 10Jforrester: jjb: Move oojs-ui-docker-publish to use node10-test-browser-php72-composer too [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) [17:49:46] (03PS3) 10Jforrester: dockerfiles: Drop npm-test-oojsui, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543895 (https://phabricator.wikimedia.org/T235570) [17:51:43] (03Merged) 10jenkins-bot: jjb: Drop oojs-ui-npm-run-jenkins-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543892 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [18:07:35] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [18:10:10] (03Merged) 10jenkins-bot: jjb: Move OOUI jobs to be withthe others in oojs.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/543893 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [18:10:14] (03Merged) 10jenkins-bot: jjb: Move oojs-ui-docker-publish to use node10-test-browser-php72-composer too [integration/config] - 10https://gerrit.wikimedia.org/r/543894 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [18:12:10] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations, and 2 others: Prepare Phame to support heavy traffic for a Tech Department blog - https://phabricator.wikimedia.org/T226044 (10srodlund) Hey all -- I am currently seeking some answers to some ba... [18:14:39] 10Phabricator: Bundle notifications from Phab - https://phabricator.wikimedia.org/T235789 (10Waddie96) [18:15:03] 10Phabricator: Bundle notifications from Phab - https://phabricator.wikimedia.org/T235789 (10Waddie96) [18:16:18] 10Phabricator: Bundle notifications from Phab - https://phabricator.wikimedia.org/T235789 (10Waddie96) [18:16:36] (03PS3) 10Jforrester: jjb: Move jobs over to php70-inherited images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [18:34:33] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Drop npm-test-oojsui, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543895 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [18:36:15] (03Merged) 10jenkins-bot: dockerfiles: Drop npm-test-oojsui, unused [integration/config] - 10https://gerrit.wikimedia.org/r/543895 (https://phabricator.wikimedia.org/T235570) (owner: 10Jforrester) [19:41:19] (03PS4) 10Jforrester: jjb: Use subsidiary images changed by move to versioned PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [19:41:21] (03PS2) 10Jforrester: dockerfiles: Drop unversioned PHP images, replaced by versioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/540694 [19:41:23] (03PS1) 10Jforrester: dockerfiles: Rename composer, composer-package, composer-test to be PHP-versioned [integration/config] - 10https://gerrit.wikimedia.org/r/543938 [19:41:25] (03PS1) 10Jforrester: jjb: Ensure jobs use versioned images of php70, not unversioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/543939 [19:41:33] OK, I think I just might have it. [19:41:38] (Oy.) [19:42:05] (03PS3) 10Jforrester: layout: [wikimedia/fundraising/crm] Make …-composer-php70-docker a full job [integration/config] - 10https://gerrit.wikimedia.org/r/540664 (https://phabricator.wikimedia.org/T230446) [19:42:18] thcipriani https://gerrit-review.googlesource.com/c/gerrit-ci-scripts/+/241542 :( [19:42:26] (gerrit 2.15 is no longer going to be tested) [19:42:50] Fun. Good thing we're going to move to 2.16. [19:43:03] Soon™ [19:43:23] but not a great place to be anyway :\ [19:43:25] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: Drop unversioned PHP images, replaced by versioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/540694 (owner: 10Jforrester) [19:43:39] (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: Rename composer, composer-package, composer-test to be PHP-versioned [integration/config] - 10https://gerrit.wikimedia.org/r/543938 (owner: 10Jforrester) [19:43:56] (03CR) 10jerkins-bot: [V: 04-1] jjb: Use subsidiary images changed by move to versioned PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [19:44:00] yeh :( [19:44:09] (03CR) 10jerkins-bot: [V: 04-1] layout: [wikimedia/fundraising/crm] Make …-composer-php70-docker a full job [integration/config] - 10https://gerrit.wikimedia.org/r/540664 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [19:44:57] (03CR) 10jerkins-bot: [V: 04-1] jjb: Ensure jobs use versioned images of php70, not unversioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/543939 (owner: 10Jforrester) [19:45:07] thcipriani do we want to install websession-flatfile to fix websessions? [19:45:39] (03PS2) 10Jforrester: dockerfiles: Rename composer, composer-package, composer-test to be PHP-versioned [integration/config] - 10https://gerrit.wikimedia.org/r/543938 [19:45:41] (03PS2) 10Jforrester: jjb: Ensure jobs use versioned images of php70, not unversioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/543939 [19:45:43] (03PS5) 10Jforrester: jjb: Use subsidiary images changed by move to versioned PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [19:45:45] (03PS3) 10Jforrester: dockerfiles: Drop unversioned PHP images, replaced by versioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/540694 [19:46:01] (03CR) 10Jforrester: dockerfiles: Rename composer, composer-package, composer-test to be PHP-versioned (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/543938 (owner: 10Jforrester) [19:46:12] paladox: it seems like it's worth experimenting with since a larger install than us has had success [19:46:14] with it [19:46:19] ok [19:46:24] * paladox creates patch to install [19:47:55] want to get a better idea of what the rollout would be like, see how well it integrates/if there is any weirdness with ldap auth (since I'm not sure if there's anyone currently using ldap using it) [19:49:28] (03CR) 10Jforrester: [C: 03+2] dockerfiles: Rename composer, composer-package, composer-test to be PHP-versioned [integration/config] - 10https://gerrit.wikimedia.org/r/543938 (owner: 10Jforrester) [19:49:30] (03PS1) 10Paladox: Add websession-flatfile plugin [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/543940 (https://phabricator.wikimedia.org/T222472) [19:49:45] thcipriani oh, i used it once and it worked [19:49:50] this was only around april. [19:49:58] It's just like h2cache [19:50:09] (03CR) 10jerkins-bot: [V: 04-1] Add websession-flatfile plugin [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/543940 (https://phabricator.wikimedia.org/T222472) (owner: 10Paladox) [19:50:10] but only stores it per file [19:50:54] We need to remove --incompatible_disallow_load_labels_to_cross_package_boundaries=false [19:51:03] (03Merged) 10jenkins-bot: dockerfiles: Rename composer, composer-package, composer-test to be PHP-versioned [integration/config] - 10https://gerrit.wikimedia.org/r/543938 (owner: 10Jforrester) [19:51:37] (03PS2) 10Paladox: Add websession-flatfile plugin [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/543940 (https://phabricator.wikimedia.org/T222472) [19:52:14] (03CR) 10jerkins-bot: [V: 04-1] Add websession-flatfile plugin [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/543940 (https://phabricator.wikimedia.org/T222472) (owner: 10Paladox) [19:52:53] (03PS1) 10Paladox: Remove --incompatible_disallow_load_labels_to_cross_package_boundaries from gerrit image [integration/config] - 10https://gerrit.wikimedia.org/r/543941 [19:53:58] (03Abandoned) 10Paladox: Merge tag 'v2.16.11' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/533350 (owner: 10Paladox) [19:54:27] (03Restored) 10Paladox: Merge tag 'v2.16.11' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/533350 (owner: 10Paladox) [19:54:29] (03CR) 10Paladox: [C: 03+2] Merge tag 'v2.16.11' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/533350 (owner: 10Paladox) [19:55:01] (03Abandoned) 10Paladox: Merge tag 'v2.15.16' into wmf/stable-2.15 [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/533332 (owner: 10Paladox) [19:55:19] (03PS3) 10Paladox: Add websession-flatfile plugin [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/543940 (https://phabricator.wikimedia.org/T222472) [19:56:25] (03PS4) 10Paladox: Add websession-flatfile plugin [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/543940 (https://phabricator.wikimedia.org/T222472) [19:58:47] (03Abandoned) 10Paladox: Remove --incompatible_disallow_load_labels_to_cross_package_boundaries from gerrit image [integration/config] - 10https://gerrit.wikimedia.org/r/543941 (owner: 10Paladox) [20:01:44] (03Merged) 10jenkins-bot: Merge tag 'v2.16.11' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/533350 (owner: 10Paladox) [20:05:39] (03PS1) 10Paladox: Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/543947 [20:07:51] (03CR) 10Paladox: [C: 03+2] Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/543947 (owner: 10Paladox) [20:08:48] thcipriani would you be able to review https://gerrit.wikimedia.org/r/#/c/operations/software/gerrit/+/538619/-1..1 and https://gerrit.wikimedia.org/r/#/c/operations/software/gerrit/+/543940/ please? :) [20:10:30] thcipriani ohh, i think upstream are moving to use the Jenkinsfile for verification. [20:10:43] I guess that's why they are removing the ci job for -flow from both 2.14/2.15 [20:14:50] Krenair: ping :) [20:19:06] (03Merged) 10jenkins-bot: Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/543947 (owner: 10Paladox) [20:24:24] (03PS3) 10Jforrester: jjb: Ensure jobs use versioned images of php70, not unversioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/543939 [20:24:26] (03PS6) 10Jforrester: jjb: Use subsidiary images changed by move to versioned PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [20:24:28] (03PS4) 10Jforrester: dockerfiles: Drop unversioned PHP images, replaced by versioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/540694 [20:24:30] (03PS1) 10Jforrester: dockerfiles: [composer-package-php70] Fix chmod for run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/543957 [20:24:37] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [composer-package-php70] Fix chmod for run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/543957 (owner: 10Jforrester) [20:26:32] (03Merged) 10jenkins-bot: dockerfiles: [composer-package-php70] Fix chmod for run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/543957 (owner: 10Jforrester) [20:36:20] Doin't you just hate it when your mobile provider has an outage but takes them 9 hours to report it... What's worse, took them the day to fix the issue... [20:36:41] It was mostly fixed earlier [20:36:48] Most iphones just gave up registering [20:36:51] So needed a reboot [20:36:56] registering? [20:37:12] Yes [20:37:23] I presumed it was only my area that had the area [20:37:38] as the network status thing was reporting an error when typing my postcode in. [20:37:50] It did that for me, but was fine for my parents and other postcodes [20:37:57] Was working fine for people in other parts of the country, then it broke [20:38:30] Oh [20:38:41] I didn't receive that on my ipohne [20:38:49] It was just impossible to use the internet. [20:39:36] (03PS4) 10Jforrester: jjb: Ensure jobs use versioned images of php70, not unversioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/543939 [20:39:38] (03PS1) 10Jforrester: dockerfiles: [composer-test-php70] Fix chmod for run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/543961 [20:39:55] Reedy will you be trying to get compensation for it? [20:39:57] (03CR) 10Jforrester: [C: 03+2] dockerfiles: [composer-test-php70] Fix chmod for run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/543961 (owner: 10Jforrester) [20:40:13] My phone had absolutely no signal [20:40:25] heh [20:40:25] They were just preparing you for Brexit. [20:40:30] lololol [20:41:26] James_F someones been missuing the 5G :P [20:41:35] *missusing [20:42:11] (03Merged) 10jenkins-bot: dockerfiles: [composer-test-php70] Fix chmod for run.sh [integration/config] - 10https://gerrit.wikimedia.org/r/543961 (owner: 10Jforrester) [20:44:14] Reedy my phone had none till i switched the cellular off and on again. [20:45:04] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/543939 (owner: 10Jforrester) [20:45:43] (03PS7) 10Jforrester: jjb: Use subsidiary images changed by move to versioned PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [20:48:03] !log Arm eventlogging key via keyholder for beta cluster following the instructions on Wikitech for T235674 [20:48:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:48:07] T235674: Beta cluster doesn’t update since ca. 2019-10-15 21:00 UTC - https://phabricator.wikimedia.org/T235674 [20:48:27] (03Merged) 10jenkins-bot: jjb: Ensure jobs use versioned images of php70, not unversioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/543939 (owner: 10Jforrester) [20:49:43] (03PS4) 10Jforrester: layout: [wikimedia/fundraising/crm] Make …-composer-php70-docker a full job [integration/config] - 10https://gerrit.wikimedia.org/r/540664 (https://phabricator.wikimedia.org/T230446) [20:53:56] elukey: ping [20:54:17] elukey: I think I fixed the scap issue you ran into [20:56:01] (03CR) 10Jforrester: [C: 04-1] "Specifically, the repo has code that doesn't pass lint." [integration/config] - 10https://gerrit.wikimedia.org/r/540664 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [20:56:10] ^ yes, but pretty sure only until next time the keyholder is disarmed/armed again / reboot [20:56:32] hauskater: puppet reverted the key to the version without passphrase.. right [20:58:35] apparently yeah [20:59:18] hauskater: you need to replace the key in the repo labs/private [20:59:21] it hink [20:59:32] ./modules/secret/secrets/keyholder/eventlogging [20:59:46] that repo is public [21:00:35] best if we doc this on the Ticket [21:00:44] I'll take a look [21:02:42] mutante: https://github.com/wikimedia/labs-private/blob/master/modules/secret/secrets/keyholder/eventlogging [21:03:05] so I need to add a key to that file right? [21:03:15] as I did on the server? [21:04:07] yea. if the fix is to add passphrases to all keys [21:04:12] and only eventlogging key did not have one [21:04:25] then the fix is to add it and upload the new file [21:04:43] but check the other files first if they have passphrases or not? [21:05:33] mutante: when doing keyholder arm several of those didn't had one [21:05:53] and I don't think Krenair added one for mwdeploy yesterday there? [21:06:04] so... how are we doing things? [21:06:04] things i can say: [21:06:18] - in production they have passphrases (several keys have the same passhrase though) [21:06:27] so you dont have to enter a separate one for each [21:06:43] - the error message you pasted said "does it have a passphrase" [21:06:56] - you said it worked for you after you added a passphrase [21:07:28] passphrases dont really matter because the repo is not actually private [21:07:57] but you would just add some "fake" ones [21:08:05] to make the check happy and be able to deploy [21:08:43] - i dont know where you would store the passphrases as you need them to do keyholder arm in the future [21:08:53] probably wikitech [21:11:06] they're in wikitech [21:11:11] at least I used those [21:12:04] mutante: so until the next keyholder reboot we're safe apparently [21:12:51] hauskater: sounds like it because you said scap works [21:12:59] but we also saw puppet revert it [21:13:27] so yea.. until either the instance restarts or new instance or labs maintenance or somebody disarms it.. so probably soon-ish [21:13:47] https://integration.wikimedia.org/ci/view/Beta/ seems happy, which had the same issue yesterday [21:14:11] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [21:14:14] so well, I can try and add a password for that key on labs/private [21:14:17] best to update the ticket, hauskater [21:14:19] maybe tomorrow [21:14:20] add what you did to fix it [21:14:33] add the entire command [21:15:38] (03PS8) 10Jforrester: jjb: Use subsidiary images changed by move to versioned PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) [21:16:16] (03CR) 10Jforrester: [C: 03+2] "Missed one." [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [21:18:36] (03Merged) 10jenkins-bot: jjb: Use subsidiary images changed by move to versioned PHP images [integration/config] - 10https://gerrit.wikimedia.org/r/540683 (https://phabricator.wikimedia.org/T230446) (owner: 10Jforrester) [21:19:29] (03PS5) 10Jforrester: dockerfiles: Drop unversioned PHP images, replaced by versioned ones [integration/config] - 10https://gerrit.wikimedia.org/r/540694 [21:21:14] (03Abandoned) 10Jforrester: layout, jjb: [wikimedia-fundraising-crm] Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514057 (https://phabricator.wikimedia.org/T223348) (owner: 10Jforrester) [21:22:39] mutante: https://phabricator.wikimedia.org/T235674#5585184 [21:22:43] sounds good? [21:23:36] hauskater: yes, sounds good [21:24:06] I forgot the chmod process though [21:24:21] not quite related to the issue which was a passwordless keypair [21:24:46] I saw thcipriani 's SAL entries fixing this in the past, as well as twentyafterfour 's I think [21:26:37] (03PS5) 10Jforrester: layout: [integration/docroot] Switch from php56 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/516570 [21:26:39] (03PS7) 10Jforrester: layout: [SmashPig] Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (https://phabricator.wikimedia.org/T224906) [21:26:41] (03PS9) 10Jforrester: jjb: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 (https://phabricator.wikimedia.org/T224906) [21:26:43] (03PS9) 10Jforrester: dockerfiles: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) [21:31:54] thcipriani: re passwordless keyholder, no idea [21:32:15] * thcipriani looks [21:32:57] thcipriani: thanks, I hope I've not broken anything [21:33:35] hauskater: if you got stuff working, sounds like you didn't break anything :P [21:33:36] :) [21:35:00] adding a passphrase to eventlogging key made it work [21:35:09] but the fix isnt in puppet [21:52:12] no keypair in labs/private appears to have a password so I'd be inclined to believe thcipriani 's comment that at some point keyholder operated with passwordless keypairs [21:52:32] I trust you guys will know better [21:53:49] seems to be controlled via profile::keyholder::server::require_encrypted_keys [21:54:27] but it doesn't seem like that was ever set (afaict) in hieradata for beta [21:54:37] thcipriani: https://github.com/wikimedia/puppet/blob/17aa5c079655ab402a9bd5ea56a6b262e7d7b317/modules/profile/manifests/keyholder/server.pp [21:54:45] although it may have been set in one of the many places that hieradata isn't very grepable [21:55:06] I see Hiera on Wikitech and on Horizon [21:55:16] and I think I remember that setting on Horizon thcipriani [21:55:28] I could log in and take a look [21:55:33] but I cannot edit it [21:55:50] yeah, there are a couple places in horizon it could exist [21:56:00] possibly also one on wikitech (if that one still is a thing) [21:57:17] require_encrypted_keys: hiera('profile::keyholder::server::require_encrypted_keys', 'yes'); agents: hiera('profile::keyholder::server::agents', {}) [21:57:33] but then [21:57:35] keyholder::require_encrypted_keys: 'no' [21:57:39] lol [21:57:53] where do you see the keyholder::require_encrypted_keys: 'no' ? [21:59:09] thcipriani: on https://horizon.wikimedia.org/project/puppet/ [21:59:13] for deployment-prep [21:59:24] well, that explains it [21:59:39] looks like the move to using profile::keyholder changed how the "no" was set. 5bd46fb369a1c25f8bdf249a64786e804ffb0e0f adds back the capability to turn off the requirement in hiera, but changes the key you need to set to 'no' to have that take affect [21:59:46] and we didn't update beta [21:59:51] * thcipriani makes puppet patch [22:01:31] thcipriani: so we need to specify 'no' for each key we want to be passwordless now then? [22:02:10] nah, I think we just need to set profile::keyholder::server::require_encrypted_keys: no [22:02:19] for beta [22:02:41] that's done on Wikitech or Horizon? [22:02:52] if you do it based on role or prefix it won't happen again when the host name changes [22:02:57] too many config spread around [22:03:17] hauskater: horizon -> prefix puppet / project puppet [22:03:20] probably [22:05:49] I think it'd be nice to just set it in hieradata/labs/deployment-prep/common.yaml so it's easier to grep for :) [22:07:01] I'm not sure where we should be adding it [22:07:06] https://gerrit.wikimedia.org/r/544064 [22:07:07] We have Hiera data on Wikitech [22:07:10] on Horizon [22:07:15] and now on Gerrit as well [22:07:19] Jesus... [22:07:21] ^ [22:07:39] it's impossible to find anything has been my experience :) [22:08:16] we're hopefully killing some of those methods soon [22:08:31] after that is merged, then sudo keyholder arm will not complain about all etc/keyholder.d keys not having a password, and will load them all? [22:08:34] including operations/puppet.git hieradata/labs [22:08:38] anyway I think the whole allowing unencrypted keys thing is a distraction from the real problem [22:08:46] all our actual keys have passwords [22:09:14] so going forward hieradata will only be in horizon? [22:10:38] authored in horizon and copied to wikitech or a separate git repo [22:10:43] automatically [22:10:49] that'd be great [22:11:04] anyway [22:11:08] Good night people [22:11:52] thcipriani, here's an easy way to tell that you have the wrong key [22:12:03] /etc/keyholder.d/deploy_service is not an acceptable key. Is it an RSA or ED25519 key with passphrase? [22:12:04] okay so: [22:12:09] 5dc50f6842d4ec28ab4c4a61973f04ac99407edad94d00f0e2bcadc147c13068 /etc/keyholder.d/deploy_service [22:12:28] Seems fine right? Nope: [22:12:34] alex@alex-laptop:~/Development/Wikimedia/Labs-Private (master)$ sha256sum modules/secret/secrets/keyholder/deploy_service [22:12:35] 5dc50f6842d4ec28ab4c4a61973f04ac99407edad94d00f0e2bcadc147c13068 modules/secret/secrets/keyholder/deploy_service [22:12:43] Something has gone very wrong for those to match. [22:13:42] same for eventlogging [22:14:02] same for phabricator [22:14:05] I used a different method [22:14:05] the 2nd thing is the sha256 of the private ssh key, what was the first hash? [22:14:23] When keylogger complained about wrong password I went to see if eventlogging indeed had a password [22:14:33] sudo cat etc/keylogger.d/eventlogging [22:14:33] sha256sum of the private ssh key on the box [22:14:47] no `ENCRYPTED` title == no password [22:15:06] I just added a password for the private key, arm keylogger and voilà [22:15:55] As I said before this is a distraction from the problem. [22:17:15] We should not be using keys in that public repository in deployment-prep. [22:20:38] we've had the understanding that labs/private is (counterintuitively) public for as long as I've been around [22:22:40] Okay good, so we understand therefore that something is going wrong if a deployment-prep is trying to use this key, regardless of whether we encrypt it or not? [22:27:57] I'm not sure I follow you. I agree that at a surface level "private" keys in repos names private being actually public and actually used seems like a bad thing; however, these keys are of limited utility without shell access to beta in which case these keys give you a limited subset of the capabilities you've already been granted. [22:29:13] thcipriani: fwiw, I'm not sure why are we importing into Diffusion refs/notes/review commits? [22:29:16] The current IP whitelist of hosts allowed to log in to a given service account is viewed as a hack that is only necessary in labs. [22:29:52] I don't want it to be difficult to get rid of in the theoretical future on the basis that we trust compromised keys. [22:29:59] We have keys set up properly for this. [22:31:14] I can agree we shouldn't trust the keys in beta for much of anything :) [22:31:17] The fix here is to ensure the correct keys get used, not to weaken security settings or encrypt public data. [22:34:18] In fact I might just upload a commit to labs/private.git that replaces these files with 'SNAKEOIL' like we do for certs. [22:49:42] Krenair: if you do that you will break at least one project where I use scap3 :/ It would make it impossible to use scap3 without a project local puppetmaster which I guess is not a huge burden on top of the already convoluted dance that using it requires... [22:53:12] This is sounding increasingly scary [22:54:46] should we be looking into using ssh certificates instead of keys? [22:54:55] sorry I'm late to the discussion [23:08:53] I don't know if that would solve anything here? [23:12:23] Still trying to fully understand what is the issue [23:12:36] it's just that we have private keys in the "private" labs repo? [23:12:49] and that might mislead people because it's not actually private? [23:13:43] partially, and the fact that our servers have actually read in those keys and treated them as actual key material to be used [23:15:10] but only virtualized test servers within labs, right? what problem does it cause? [23:15:37] sorry I mean within deployment-prep only, right? [23:19:42] well I assumed only really deployment-prep might accidentally try to use these keys but bd808 said there's one where it's done knowingly [23:20:40] so you're worried about potentially reusing a key that's also used elsewhere? [23:21:21] one problem is that if you generate a keypair, put the public part on your account's list of authorised keys, and post the private part on the public internet, anybody can log into your host [23:21:42] only if that host is accessible to them [23:21:59] there are multiple layers of access required here [23:22:04] we run in labs with SSH allowed from the bastions [23:22:39] sure, but not with those keys granting authn on the bastions [23:22:43] our only defence against this right now that I'm aware of is an IP whitelisting thing we have to do to allow service accounts to be SSHed into that would not normally be allowed due to the labs LDAP group membership check [23:22:59] so you need a "good" key for the bastion and then could use the "bad" key to go somewhere else [23:23:05] we don't trust everyone on the bastions [23:23:14] they are for these purposes the public internet too [23:24:28] Krenair: so you're worried that it could allow access to a user's personal machine due to them authorizing the key locally? [23:24:39] I follow your threat modeling, but I do not see the practical issue of abuse yet [23:24:40] no this has nothing to do with users personal machines [23:24:41] * twentyafterfour wouldn't authorize a key that I planned to use on deployment-prep [23:24:52] oh [23:25:40] These keys would allow someone to change the deployed code on a scap3 managed deploy inside a Cloud VPS project. Agreed. [23:26:25] But only if they first could get past the jump host (bastion) [23:26:34] and then know where they wanted to go [23:26:45] how about actually adding a passphrase to the keys (the same one) instead of that Hiera setting to skip it.. then store the passphrase in an actually secret place.. pwstore? [23:26:50] I'm not willing to trust the labs network. [23:26:56] and then what exactly can they do to harm a service in a non-trusted environment? [23:27:55] mutante, you're talking about encrypting a key that is already posted publicly... [23:27:56] mutante: and then expose that passphrase in deployment-prep (large user base) and any other project that needs the keys [23:27:57] so you are back to pretty much the same place [23:28:14] labs secrets should not be going anywhere near ops' pwstore [23:28:24] i never said ops' [23:28:30] for tenants anyway [23:28:42] I don't believe we have a pwstore for deployment-prep [23:29:14] i thought the problem vs "public internet vs. just users of deployment-prep" [23:29:20] The existing solution for this is files stored on the deployment-puppetmaster [23:29:24] Which is fine [23:29:54] The problem is it seems at some point we stopped using those files and reverted to using the snakeoil keys [23:30:49] Krenair: is the difference in your mind that with a project local puppetmaster and separate keys there the exposure is to a smaller number of people? [23:30:52] encrypting those snakeoil keys does nothing except sneak it past keyholder's checks for unencrypted keys, which as I said earlier is a distraction from the real problem here [23:33:28] make new keys, encrypt them, upload encrypted keys to labs/private ? [23:33:35] We already have this! [23:33:40] That's the crazy thing about this whole discussion. [23:33:48] We have separate keys. [23:33:52] They are in labs/private. [23:34:04] They were encrypted with passphrases stored alongside the files also in labs/private [23:34:34] This is the labs/private directory on the puppetmaster itself [23:34:37] I.e. not Gerrit. [23:35:23] But for some reason that I don't know yet, at some point the cluster started using the keys that were uploaded to gerrit [23:36:26] Maybe it was some change of path thing [23:37:08] Maybe we lost some commit from our labs/private cherry-pick list that copied files to where they actually get picked up etc. [23:37:10] I'm not sure [23:38:12] got it. so the "actually private labs/private". but it wouldnt be necessary to use that for the keys, you could as well use public labs/private. the problem is just where to store the passphrase. [23:38:20] But instead of fixing that the response I've got is people questioning why we bother with actually private SSH keys instead of a) trusting the labs network or b) relying on the IP whitelisting alone that I don't really like in the first place [23:39:25] I was mostly responding to the idea of nuking the keys in labs/private.git. I don't care what deployment-prep does [23:39:51] mutante, So your proposal is to take the encrypted, actually private, key and upload it to gerrit, just leaving the passphrase in our cherry-picks? [23:40:21] we wouldn't have the copy the key itself around in our labs/private cherry-picks, it'd already be there? [23:41:26] I feel like if we did that it'd be more likely for someone to copy it to a separate project. I'm not really sure this is necessary when we could just log into the puppetmaster and run a few cp commands and commit the fix. [23:42:01] Krenair: yea, basically like it is now except the keys actually have a passphrase and there is _some_ place to keep the passphrase where people who run keyholder arm can get it from [23:42:15] We already did this for mwdeploy [23:42:33] mutante, we already have that though [23:43:01] This stuff all lives at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin [23:43:09] it seems closer to how it's in production and you avoid that extra Hiera setting to disable passphrases [23:43:37] we don't need any hiera setting to disable requiring passphrases - the proper keys have passphrases [23:43:50] The mystery is why the proper keys are not getting used [23:46:46] It's probably something to do with the directory being called tin or something