[03:55:10] <Raymond_Ndibe>	 Heads up everyone: I manually executed the scheduled pipeline here https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/pipeline_schedules to force auto-update pre-commit for our repositories. The reason for this can be found here https://phabricator.wikimedia.org/T372601. Basically the last run of that scheduled pipeline committed a version of golangci-lint that wasn't working for our CI and as a result all our 
[03:55:10] <Raymond_Ndibe>	 golang repositories were failing CI. Incase you wake up to a lot of gitlab MR email activities all of that was my fault, for good reason
[14:40:59] <btullis>	 dhinus: I added you to a patch about ceph reimaging, since we were talking about it recently. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1064388
[14:42:16] <btullis>	 We want to test out removing `$(hostname)-vg`and associated LVM signatures while reimaging, but leaving all other LVM signatutes untouched.
[14:42:54] <btullis>	 As the patch stands, this would affect cloudcephosd* hosts as well, but we could exclude them if you prefer.
[14:49:04] <dhinus>	 thanks! IIRC we actually tried following your vg schema for a cloudcephosd host that was recently reimaged
[14:49:45] <dhinus>	 it would be good to have a shared partition list/names
[14:50:50] <dhinus>	 andrewbogott: I was chatting with arnaudb who is playing with some cloudvps instances in the mariadbtest project
[14:51:24] <dhinus>	 arnaudb tried using the "keypair" feature, but that seems to have interfered with the standard auth
[14:51:48] <dhinus>	 so now I can ssh as root to db-mariadbtest-1 but not as a standard user
[14:52:07] <andrewbogott>	 Yeah, ideally that panel would appear for unpuppetized images but vanish for puppetized base images.
[14:52:09] <dhinus>	 do you know if there's a quick way to reset it? or maybe it's easier to delete and recreate that vm
[14:52:27] <andrewbogott>	 But the keypair thing really should break standard auth. Is puppet failing there?
[14:52:41] <andrewbogott>	 sorry, *really should not break"
[14:54:04] <dhinus>	 yes puppet is failing, but probably for unrelated reasons
[14:54:23] <dhinus>	 "Could not request certificate: The certificate retrieved from the master does not match"
[14:54:35] <dhinus>	 that's probably the local puppet config that's broken
[14:54:41] <andrewbogott>	 If it didn't get a proper puppet run ever then auth won't get set up
[14:54:48] <dhinus>	 yep that makes sense
[14:55:11] <andrewbogott>	 so probably the keypair thing is unrelated
[14:55:18] <andrewbogott>	 unless y'all have tested it both ways
[14:55:37] <dhinus>	 the first puppet run worked
[14:55:42] <dhinus>	 but then something broke it
[14:55:57] <andrewbogott>	 probably it has a project config that switches it to a different puppetserver
[14:56:03] <arnaudb>	 (or someone)
[14:56:05] <arnaudb>	 (me)
[14:56:07] <dhinus>	 :D
[14:56:31] <andrewbogott>	 btw my personal key works fine on that VM
[14:56:34] <dhinus>	 yep I think the attempt to link it to the local puppetserver didn't work and things started failing
[14:56:41] <andrewbogott>	 so pam/sssd seems to be working fine
[14:56:42] <arnaudb>	 mind that: instead of debugging, if somebody has a magic formula to reset everything to 0 on this project, it'd be good enough as I was only starting to tinker when I broke things
[14:57:05] <andrewbogott>	 arnaudb: what do you mean by 'reset everything to 0'?
[14:57:06] <arnaudb>	 (don't want to make you waste time on empty vMs x)
[14:57:14] <dhinus>	 ah now my one works as well, maybe I just didn't wait long enough after adding myself to the project!
[14:57:38] <dhinus>	 arnaudb: can you try if you can ssh now to db-mariadbtest-1.mariadbtest.eqiad1.wikimedia.cloud
[14:57:39] <arnaudb>	 andrewbogott: I meant deleting the cumin/puppet nodes
[14:57:53] <andrewbogott>	 arnaudb: can you not?
[14:58:16] <arnaudb>	 I was unsure it was going to fix my issue with the ssh key as it seemed to be linked to the key I was not able to delete 
[14:58:34] <arnaudb>	 so I tried to not break it further before being able to assess where I did something wrong
[14:58:45] <arnaudb>	 dhinus: Connection closed by UNKNOWN port 65535 I'm being dropped
[14:59:01] <andrewbogott>	 It's true that there's no way to remove a keypair from your account, but you can certainly create new VMs without the keyapri
[14:59:05] <andrewbogott>	 *keypair
[14:59:07] <dhinus>	 arnaudb: and does it work with cloud-cumin-01.mariadbtest.eqiad1.wikimedia.cloud ?
[14:59:13] <andrewbogott>	 and in any case, I don't think that's related to what's happening
[14:59:37] <arnaudb>	 andrewbogott: ack, dhinus: I can! yay
[15:01:07] <dhinus>	 in db-mariadbtest-1 I see "fatal: Access denied for user arnaudb by PAM account configuration [preauth]"
[15:01:28] <dhinus>	 I wonder if ssh is trying to auth with a different key
[15:02:51] <arnaudb>	 it should be trying successively the available ones, no?
[15:03:16] <dhinus>	 yep unless it matches one that was the one managed by the openstack keypairs... but that's a long shot
[15:03:38] <dhinus>	 andrewbogott: is it expected that "openstack keypair list --project mariadbtest" is hanging without returning results?
[15:03:48] <andrewbogott>	 You should definitely delete and recreate that VM rather than being curious
[15:03:58] <andrewbogott>	 I don't see evidence that puppet has ever worked properly there
[15:04:02] <dhinus>	 andrewbogott: +1
[15:04:04] <arnaudb>	 ack, will try that tomorrow
[15:04:12] <arnaudb>	 thanks for the help <3
[15:05:33] <andrewbogott>	 + try a new hostname, there might be a stale cert for that name on the puppetserver
[15:16:20] <dhinus>	 btullis: I left a comment in the task, but I'm probably not understanding the full reimage+prepare process
[15:17:39] <dhinus>	 I think your patch is fine to merge anyway
[15:27:26] <btullis>	 Thank you kindly.
[17:41:34] <andrewbogott>	 slyngs: regarding your recent email, is 'idm-test.wikimedia.org' backed by prod ldap or codfw1dev/labtest ldap?
[18:36:01] <andrewbogott>	 dhinus: are you still around? I'm getting an alert about toolsdb
[18:39:27] <dhinus>	 checking
[18:40:37] <dhinus>	 I cannot ssh to either tools-db-1 or tools-db-3
[18:40:43] <andrewbogott>	 I started a migration to another cloudvirt just in case that matters
[18:41:03] <andrewbogott>	 hm, but db-3 is on a different cloudvirt already
[18:41:44] <dhinus>	 I have maybe ssh problems, can you ssh to any instance?
[18:41:57] <andrewbogott>	 a different VM on that host seems to work fine
[18:42:08] <andrewbogott>	 e.g. deployment-ms-be08.deployment-prep.eqiad1.wikimedia.cloud
[18:42:34] <dhinus>	 ok now I can ssh to tools-db-3
[18:42:41] <andrewbogott>	 migration finished and now I can reach -1
[18:42:45] <andrewbogott>	 hmmmm
[18:43:10] <dhinus>	 I can see tools-db-3 logged a "lost connection" to -1
[18:43:22] <andrewbogott>	 "Aug 21 18:42:34 tools-db-1 kernel: [4760409.912374] mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600"
[18:43:25] <andrewbogott>	 on -1
[18:43:54] <andrewbogott>	 so... why is that happening on different VMs in the same week?
[18:44:01] <dhinus>	 yep seems weird
[18:44:04] <dhinus>	 I'm restarting mariadb on -1
[18:44:10] <andrewbogott>	 ok
[18:44:52] <dhinus>	 this runbook is up to date in case I'm not around: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState#MariaDB_process_killed_by_OOM_killer
[18:45:01] <dhinus>	 it's also linked from the alert in alertmanager
[18:46:06] <andrewbogott>	 I don't see anything in the neutron or nova logs that would explain this
[18:46:12] <andrewbogott>	 so /maybe/ it's just coincidental ooms
[18:47:27] <andrewbogott>	 Resolved for now, right?  I'm going to clear alert manager
[18:47:31] <dhinus>	 replication of toolsdb resumed automatically after I restarted the primary
[18:47:38] <dhinus>	 so all looks good, alert manager is already clear
[18:47:38] <andrewbogott>	 that's good :)
[18:48:34] * dhinus offline