[06:00:28] <moritzm>	 FYI, I'm rebooting bast3007 as it's currently unused
[11:35:03] <Emperor>	 anyone else just get kicked off all our invite-only channels?
[11:35:22] <marostegui>	 not  me
[11:35:53] <marostegui>	 Emperor: actually I just did XD
[11:36:16] <Emperor>	 I seem to be able to just rejoin...
[11:36:35] <marostegui>	 I have the rejoin enabled, so I didn't notice until I checked the bouncer logs, my client doesn't show it
[11:38:28] <cezmunsta>	 Emperor: me too, also able to rejoin though
[11:39:59] <p858snake|cloud>	 chanserv was on the otherside of a netspilt
[14:38:17] <inflatador>	 klausman it's long overdue, but I finally took your advice and made some tmuxp dashboards! https://gitlab.wikimedia.org/repos/search-platform/es-maint-viewer
[15:12:21] <klausman>	 ooh, neat
[15:27:05] <inflatador>	 yeah, a good excuse to work in some pyinfra as well ;P
[15:53:30] <Evilham>	 👀
[17:52:26] <rzl>	 andrewbogott: https://phabricator.wikimedia.org/T427081 fyi
[17:52:33] <rzl>	 not sure if that officially gets routed to you or to I/F :)
[17:53:12] <andrewbogott>	 probably still me, alas.  Thanks
[17:55:11] <rzl>	 I was going to say, I thought we had an alert for that, but I do actually see it firing since 2026-05-18, https://alerts.wikimedia.org/?q=alertname%3DProbeHttpFailed&q=team%3Dsre&q=%40receiver%3Ddefault
[18:02:16] <andrewbogott>	 predictably, it's just that / is filled up
[18:43:23] <andrewbogott>	 is pwstore working for people today? And if so, which keyserver are you using? (I see riccardo's key as expired and can't get it to refresh from anywhere)
[18:44:47] <mutante>	 andrewbogott: we gave up on the idea to use keyserver. the keys should all be in the ./keys dir inside the repo
[18:45:14] <mutante>	 do you just need to read or also write to it today?
[18:45:29] <andrewbogott>	 both
[18:45:40] <andrewbogott>	 well, it can wait if need be
[18:46:07] <andrewbogott>	 So that means that the key is expired in git and needs to be refreshed by riccardo?
[18:46:30] <mutante>	 to read a file you can just do "gpg -d <file>" without using pws.rb ed
[18:46:43] <mutante>	 enter your own GPG passphrase and you got cleartext
[18:46:54] <mutante>	 to write.. yes, needs to be update in the repo 
[18:47:06] <mutante>	 or one of the 4 people who can edit and resign the .users file can remove it 
[18:47:56] <mutante>	 to write temporarliy you can also "gpg -e" it directly with GPG for one or multiple other keys and leave it in the repo
[18:48:06] <mutante>	 until the key is updated
[18:48:16] <andrewbogott>	 ok, I will just find something else to do for today :) thank you!
[18:48:24] <mutante>	 ;)
[18:50:13] <rzl>	 andrewbogott: fwiw, I just tried "git pull" followed by "gpg --import keys/*" followed by "pws ed wikitech-static" and it works for me
[18:50:29] <andrewbogott>	 dammit
[18:50:33] <andrewbogott>	 ok, trying again :)
[18:50:37] <moritzm>	 or simply "pws update-keyring"
[18:50:45] <rzl>	 oh even better
[18:51:10] <andrewbogott>	 no, pws update-keyring fails because of the expired key
[18:51:13] <andrewbogott>	 that was where I gave up
[18:51:20] <andrewbogott>	 gpg --import seems to work though
[18:51:20] <rzl>	 (I see Riccardo's key updated in a3187ac2, on March 16)
[18:51:28] * andrewbogott updates the docs
[18:54:50] <moritzm>	 that can't be true, pws update merely concatenates the keys
[18:55:05] <moritzm>	 but obviously you need to run git pull to update keys
[18:55:12] <moritzm>	 it's been like this for five years or so
[18:56:43] <moritzm>	 simply run a "git pull", then "pws update-keyring" and you can update secrets again
[18:57:50] <andrewbogott>	 moritzm: here is what happened when I did what you suggest (this starts after a fetch and reset --hard)
[18:58:04] <andrewbogott>	 https://www.irccloud.com/pastebin/ixa2TYS5/
[18:58:22] <andrewbogott>	 I am unblocked for now, just fyi
[18:58:56] <moritzm>	 ok
[19:10:04] <andrewbogott>	 rzl: before I document... can you grab aws-wikitech-static.pem (which I just put in pwstore) and confirm that you can use it to ssh to admin@ec2-3-126-27-158.eu-central-1.compute.amazonaws.com ?
[19:10:21] <rzl>	 sure, let's see
[19:10:45] <andrewbogott>	 you may or may not have to strip the access header first, not sure
[19:14:14] <rzl>	 andrewbogott: do you have the ssh fingerprint posted anywhere?
[19:17:17] <andrewbogott>	 ssh-ed25519 SHA256:Nk1+oCpzZeKIs/krn8e+W/TbUl64q3CC+g/v/XnT6mQ
[19:17:27] <andrewbogott>	 not really useful to publish anywhere since the VM is largely ephemeral
[19:17:48] <andrewbogott>	 I mean, it /may/ persist but gets rebuild anytime there's a config change
[19:18:07] <rzl>	 fair enough
[19:18:10] <rzl>	 yes that works for me
[19:18:38] <rzl>	 I did have to decrypt and save it, which isn't the usual pws workflow, but as long as that's expected
[19:18:38] <andrewbogott>	 great, thanks for testing, I will point at that key on https://wikitech.wikimedia.org/wiki/Wikitech-static
[19:19:15] <andrewbogott>	 It's definitely what I expected but if there's a slicker way to distribute it I'm happy to adjust.
[19:19:30] <andrewbogott>	 I guess it could be installed somewhere by puppet
[19:19:35] <rzl>	 (the access header wasn't an issue, everything before "-----BEGIN RSA PRIVATE KEY-----" is ignored)
[19:19:40] <andrewbogott>	 cool
[19:20:09] <andrewbogott>	 The real sustainability issue here is that as far as I know no one is left at the org who can actually issue new AWS accounts.
[19:20:14] * andrewbogott hopes that isn't actually true
[19:20:43] <andrewbogott>	 I could add my personal login creds to pwstore but that's not ideal
[19:42:47] <inflatador>	 Looks like paramiko is choking on those `root@gitlab1001` comments in https://config-master.wikimedia.org/known_hosts . I'm gonna file a ticket for releng to change these, but if anyone has a different opinion LMK
[19:46:27] <inflatador>	 upon further review, it looks like is the extra space between the first 2 fields that paramiko doesn't like