[00:38:24] I wonder at what point WMCS will decide bastions are too open and make everyone tunnel in through WireGuard [00:54:13] security is not everything [00:55:27] If the goal already is to facilitate access to a larger-than-usual number of people, I don't think it provides meaningfully better security over a sufficiently locked down bastion [10:50:06] hihi, I'm looking at T424813 and puppet is failing due to `Error: /Stage[main]/Puppet::Agent/Package[puppet]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install puppet' returned 100` (`E: Packages were downgraded and -y was used without --allow-downgrades.` as it wants to downgrade `ruby-concurrent`) - O [10:50:06] T424813: [Cloud VPS alert][commtech] Puppet failure on pageviews02.commtech.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T424813 [10:50:13] *I'm not entirely sure how to fix this [10:55:12] TheresNoTime: does a plain `sudo apt install puppet` help? [10:55:52] taavi: trying that now.. [10:57:25] taavi: yup, that fixed it, thank you :) [10:57:49] now I'm curious how that instance got in that state [10:58:03] TheresNoTime: any idea when it was created? [10:58:41] I can check, but fairly sure it wasn't that recent [10:59:40] taavi: 5 weeks ago [11:06:35] huh [14:48:49] Is it expected that a puppet failure on a newly-provisioned WMCS VM will prevent SSH login? [14:54:05] likely, yes, since puppet provisions the keys and sssd setup [14:54:39] if you have a project-wide or prefix-wide puppet config it's worth bootstrapping the VM without it and then applying once it's up so you can see what's happening [14:55:13] if you /don't/ have that and puppet is still failing on bootstrap then I want to know about it [15:00:56] Yeah, I can cheat a bit with my SRE powers and get in on console too [15:01:42] Might be worth moving the login stuff into cloud-init vendor data. Although I admittedly have no idea what the expectations are in beta [15:21:01] you can inject an ssh key with cloud-init, in a pinch. Actually getting the ldap integration into cloud-init would be A Project [16:01:15] Oh yeah, that's a good idea. I'll try the SSH key thing if I can't get it going on console. I don't really like making a bunch of changes to the project puppet stuff [16:05:48] !log admin cleaning up stray broken osbpo references on VMs, e.g. /etc/apt/sources.list.d/openstack-dalmatian-bookworm.sources [16:05:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL