[06:26:49] <moritzm>	 FYI, I'l reboot puppetserver1002/2002 in approx. 10 minutes. best to hold off puppet merges until these are done (otherwise you'll run into timeouts)
[06:46:04] <moritzm>	 and these are back, puppet merges can resume
[08:31:45] <elukey>	 bd808: thanks!
[08:34:50] <elukey>	 hey folks, I am going to proceed in a few to rollout the new debmonitor's pki intermediate, as outlined in https://phabricator.wikimedia.org/T420993#11812923
[08:35:05] <elukey>	 I'll need to disable puppet fleetwide for 10/15 mins
[08:35:17] <godog>	 good luck
[08:41:10] <elukey>	 thanks <3
[08:41:15] <elukey>	 all right disabling puppet
[08:53:25] <elukey>	 merged private and public keys in puppet private/public
[08:53:33] <elukey>	 now I am going to run puppet on the pki hosts
[08:54:24] <moritzm>	 +1
[09:02:32] <elukey>	 running on debmonitor1003
[09:04:45] <moritzm>	 the refresh looks good and I've just tested a debmonitor submission on sretest1005
[09:05:11] <moritzm>	 worked just fine: https://debmonitor.wikimedia.org/hosts/sretest1005.eqiad.wmnet (inetutils-telnet is no longer shown as upgradeable)
[09:06:01] <moritzm>	 debmonitor on 1003 was also correctly restarted during the cert refresh
[09:06:57] <elukey>	 I restarted it manually in this case, from systemctl status it didn't seem to have been during the puppet run
[09:07:24] <moritzm>	 ah, ok.I had only looke at the uwsgi lifetime
[09:08:10] <elukey>	 moritzm: how are the certs handled? I see that we also have envoy with a discovery cert, but I guess that part is only for the UI right? The debmonitor server cert is exposed directly by uwsgi?
[09:08:37] <moritzm>	 indeed, envoy is only for the web UI
[09:10:31] <moritzm>	 hmmh, actually after a full restart of apache on debmonitor1003 I'm getting a cert error with debmonitor-client now
[09:10:32] <elukey>	 ah no ok not uwsgi, but httpd
[09:10:50] <elukey>	 that needs to be restarted
[09:11:09] * volans around if you need a hand
[09:12:46] <moritzm>	 https://paste.debian.net/hidden/8cad24b6
[09:14:19] <elukey>	 lovely
[09:14:40] <elukey>	 so is the leaf + intermediate ok on sretest1005?
[09:14:51] <elukey>	 maybe the debmonitor-client needs to be restarted?
[09:15:06] <volans>	 it's not a daemon
[09:15:16] <elukey>	 okok, it runs as timer 
[09:15:31] <elukey>	 or hook, don't recall
[09:15:38] <volans>	 both
[09:15:50] <volans>	 the content of /etc/debmonitor/ssl is correct?
[09:16:56] <elukey>	 in theory https://paste.debian.net/hidden/8cad24b6 seems complaining about the debmonitor server cert right?
[09:17:36] <volans>	 it seems that the client can't verify the validity of the server cert 
[09:18:16] <volans>	 so if sretesdt1005 has the root CA and debmonitor is sending back both the leaf cert and the intermediate it should work, let me check something
[09:18:21] <moritzm>	  /etc/debmonitor/ssl/debmonitor__sretest1005_eqiad_wmnet.pem is updated and looks alright to me
[09:19:10] <moritzm>	 all the certs there I mean
[09:19:53] <moritzm>	 elukey:  debmonitor can simply be run on the commandline, just running "sudo debmonitor-client" is enough
[09:20:49] <volans>	 SSL handshake has read 1712 bytes and written 426 bytes
[09:20:49] <volans>	 Verification error: unable to verify the first certificate
[09:21:04] <volans>	 with
[09:21:04] <volans>	 openssl s_client -connect debmonitor.discovery.wmnet:443 -CApath /etc/ssl/certs -servername debmonitor.discovery.wmnet
[09:21:21] <volans>	 the server is sending only the leaf I think
[09:21:40] <elukey>	 mmm lemme check
[09:21:57] <volans>	 I see only one cert in Server certificate from that command
[09:22:40] <elukey>	 elukey@debmonitor1003:~$ sudo cat /etc/debmonitor.conf 
[09:22:40] <elukey>	 [DEFAULT]
[09:22:40] <elukey>	 server=debmonitor.discovery.wmnet
[09:22:40] <elukey>	 cert=/etc/debmonitor/ssl/debmonitor__debmonitor1003_eqiad_wmnet.pem
[09:22:40] <elukey>	 key=/etc/debmonitor/ssl/debmonitor__debmonitor1003_eqiad_wmnet-key.pem
[09:23:01] <volans>	 I think we should use debmonitor__debmonitor1003_eqiad_wmnet.chained.pem
[09:23:21] <elukey>	 I agree, but how did it work till now?
[09:23:31] <elukey>	 the clients must have had the intermediate
[09:23:34] <volans>	 the old intermediate is part of ca-certificates?
[09:23:39] <elukey>	 nope
[09:23:42] <volans>	 or was directly generated from the root?
[09:23:52] <volans>	 IIRC it was a puppet cert before
[09:24:10] <elukey>	 my understanding is that we only have the regular bundle with the root pki public cert on the hosts
[09:24:14] <elukey>	 not sure about debmonitor though
[09:24:25] <moritzm>	 no, it's not part of the bundle
[09:26:45] <volans>	 wasn't the previous one generated by Puppet_Internal_CA.pem ?
[09:27:37] <Emperor>	 puppet runs still currently blocked, yes?
[09:27:48] <elukey>	 Emperor: yep sorry, we hope to unblock in a few
[09:28:04] <Emperor>	 ack, np, the cookbook will keep waiting for a while yet :)
[09:28:09] <elukey>	 volans: in theory debmonitor has its own intermediate, shouldn't use the Puppet CA
[09:28:27] <elukey>	 Emperor: sorry we just got a "surprise" that we didn't expect :D
[09:28:30] <moritzm>	 I'm currently looking at Puppet git, I was under the impression John had changed it away from a Puppet cert some years ago
[09:28:38] <volans>	 for the new one yes, I was talking how was working before, anwyay, I think if we use the chained one all works
[09:29:10] <elukey>	 volans: trying to modify the config manually so we can test on debmonitor1003
[09:29:15] <volans>	 +1
[09:31:39] <elukey>	 ok sorry the config that I pasted above was for the debmonitor *client* on debmonitor1003, the actual tls config is handled by httpd
[09:31:49] <elukey>	     SSLCertificateFile /etc/cfssl/ssl/debmonitor__debmonitor_discovery_wmnet_server/debmonitor__debmonitor_discovery_wmnet_server.pem
[09:31:49] <elukey>	     SSLCertificateKeyFile /etc/cfssl/ssl/debmonitor__debmonitor_discovery_wmnet_server/debmonitor__debmonitor_discovery_wmnet_server-key.pem
[09:31:57] <elukey>	 but same thing, no chained
[09:32:03] <volans>	 ack
[09:32:37] <volans>	 I wonder if we have to use the chained in the clients too though
[09:32:56] <volans>	 my gut would say yes
[09:33:12] <volans>	 being mutual TLS also the server needs to be able to verify the client
[09:33:50] <elukey>	 debmonitor1003 should be fixed
[09:34:31] <volans>	 elukey: changing hte client config to chained works fine
[09:34:32] <volans>	 without doesn't
[09:34:44] <volans>	 so we need the patch to also change the client's cert path
[09:34:49] <elukey>	 lovely
[09:34:56] <volans>	 that IMHO confirms that the issuer of the previous cert was in ca-certificates
[09:35:53] <elukey>	 we can check the old public for debmonitor, it should contain the data
[09:36:05] <elukey>	 anyway, looking into the puppet patch
[09:36:26] <moritzm>	 the only reference I could find in Phab was https://phabricator.wikimedia.org/T340741#9001934 by John, but he might have confused it with the discovery cert from https://phabricator.wikimedia.org/T281377
[09:40:20] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1270870
[09:41:11] <elukey>	 running pcc
[09:43:02] <volans>	 k
[09:43:04] <elukey>	 volans, moritzm https://puppet-compiler.wmflabs.org/output/1270870/8408/ (failed to post)
[09:43:09] <elukey>	 it looks good afaics
[09:43:44] <volans>	 why didn't change the client config on debmonitor1003?
[09:43:53] <volans>	 ah no it did
[09:43:54] <volans>	 my bad
[09:44:05] <moritzm>	 looks good to me
[09:44:08] <volans>	 shiop it
[09:46:06] <elukey>	 running puppet on debmonitor servers + httpd restart
[09:46:27] <elukey>	 I really can't wait to do the discovery intermediate rotation
[09:46:40] <elukey>	 it will be soo much fun
[09:46:44] * elukey cries in a corner
[09:46:58] <volans>	 ahahaha
[09:47:46] <elukey>	 all good on the server side
[09:47:53] <elukey>	 let's verify on sretest1005
[09:48:01] <moritzm>	 confirmed working on sretest1005
[09:48:17] <moritzm>	 forced a Puppet run and upgraded openssh and all properly recorded: https://debmonitor.wikimedia.org/hosts/sretest1005.eqiad.wmnet
[09:48:29] <elukey>	 moritzm: shall we test another couple of hosts just to be sure? then we can re-enable puppet 
[09:48:42] <moritzm>	 yes, let me also quickly validate on a bullseye and trixie node, I'll report back
[09:48:42] <volans>	 pick a bullseye
[09:48:49] <volans>	 great
[09:53:11] <moritzm>	 bullseye and trixie work fine, we can re-enable Puppet
[09:53:42] <volans>	 \o/
[09:55:26] <elukey>	 all right, re-enabling
[09:59:12] <elukey>	 all done!
[09:59:19] <elukey>	 Emperor: you should be unblocked
[09:59:30] <elukey>	 thanks volans and moritzm 
[09:59:53] <volans>	 anytime :)
[10:00:50] <moritzm>	 elukey: as I said before it will all be fine :-)
[10:02:02] <volans>	 ™
[10:02:08] <elukey>	 moritzm: very happy that the new puppet part worked fine, but now I am a little more hesitant for the discovery part. It will probably require a day :D
[10:02:43] <elukey>	 and I need to test the cert-manager part of k8s first, to avoid having clusters pulling certs automatically
[10:04:43] <Emperor>	 elukey: thanks :)
[10:13:21] <moritzm>	 headsup: the cumin1003 reboot/update to Cumin 6 will commence shortly, please don't start any new tmuxes/screens there
[10:18:56] <claime>	 I may have a tmux open there but nothing running on it feel free to boot me
[10:19:41] <moritzm>	 ack
[10:19:50] <moritzm>	 I'll report back when it's completed
[10:25:08] <moritzm>	 cumin1003 can be used again
[10:26:09] <moritzm>	 Riccardo just upgraded it to Cumin 6, which has been running on cumin2002 before, but since more people tend to use cumin1003 than cumin2002, if you see any issues (which are not expected at this point),please speak up
[10:26:22] <volans>	 indeed
[10:28:13] <moritzm>	 as usual, there's some WIP wmfmariadb-py packages flagged for downgrade, not opening that can of worms right now :-)
[10:28:52] <moritzm>	 these are probably test packages so I left them in their current versions
[11:57:54] <XioNoX>	 volans, claime, I'm depooling esams for the network OS upgrade (same as last tuesday hopefully with less crashloop bugs :)
[11:58:16] <claime>	 XioNoX: ack
[11:59:40] <volans>	 XioNoX: ack
[12:53:56] <volans>	 moritzm: seems that puppetserver1002 is having som eissues, we have 137 hosts with puppet failed
[12:54:00] <volans>	 https://puppetboard.wikimedia.org/nodes?status=failed
[12:54:11] <volans>	 aything currently in progress puppet wise?
[12:55:29] <volans>	 they are all timeouts or Early EOF, so potentially also network related
[12:56:00] <moritzm>	 not really, but puppetserver1002 was rebooted earlier the day and maybe some SPF is acting up after the reboot? I'll depool it for further analysis
[12:56:24] <volans>	 ack thx, I'll keep an eye on puppetboard
[12:56:30] <volans>	 to see if it improves
[12:56:36] <volans>	 (154 failed now)
[13:01:51] <moritzm>	 volans: https://gerrit.wikimedia.org/r/c/operations/dns/+/1270924
[13:02:07] <Emperor>	 //Unable to run puppet on config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production// <-- is this something that should resolve with a subsequent puppet run? [from a reimage cookbook run]
[13:02:29] <volans>	 +1ed
[13:02:43] <volans>	 Emperor: see private
[13:02:49] <volans>	 and here too
[13:03:27] <Emperor>	 ah, right, I see. I picked a bad day to be doing a bunch of reimages...
[13:03:36] <volans>	 wait for the depool and then retry
[13:04:06] <fabfur>	 confirm that now it's better (at least on lvs1019) so it should be already depooled
[13:04:21] <volans>	 fabfur: is not yet merged
[13:04:27] <fabfur>	 oh
[13:04:58] <fabfur>	 now it applied the catalog without timeouts (but can't send report to puppetserver1002: eof)
[13:07:06] <moritzm>	 puppetserver1002 is now depooled, but it'll take a little until clients requery the service record, should recover soon
[13:07:20] <moritzm>	 going to add a task for troublesooting
[13:07:46] <volans>	 thx
[13:11:07] <moritzm>	 https://phabricator.wikimedia.org/T423282
[13:12:46] <volans>	 thx, added some exceptions examples from the logs
[13:13:06] <volans>	 not sure why, the failed nodes are increasing though :/
[13:14:31] <volans>	 but all the failures I'm seeing are always towards 1002
[13:15:04] <moritzm>	 the current spike is most certainly linked to the esams maintenance as well?
[13:16:07] <moritzm>	 there's no new requests incoming to puppetserver1002, so it shouldn decrease now
[13:16:10] <moritzm>	 there's no new requests incoming to puppetserver1002, so it should decrease now
[13:16:15] <volans>	 it went from ~150 to ~280, so much more than esams :D but seems to be going down 
[13:23:18] <volans>	 back to 161 and recovering
[13:53:47] <elukey>	 hey folks, I made a list of hosts impacted by the pki discovery rollout: https://phabricator.wikimedia.org/T420993#11819672 - Please check them and see if I need to rework the categories, the idea would be to go from low to high risk when rolling out
[13:54:15] <elukey>	 the other bit is also related to a restart that may be required, if the new cert is not picked up by default
[14:01:00] <Emperor>	 I've re-imaged ms-be2068 twice now; and both times the initial reboot into a vanilla bullseye image goes fine, but once the initial puppet run has happened the system is no longer able to boot - it just keeps cycling round trying to boot from the (I think correct) disk, pausing a bit, and then rebooting again, with no sign of a GRUB menu or anything. [the cursor moves up and down a bit on an otherwise-empty display] Any ideas?
[14:02:35] <Emperor>	 (other than open a ticket with infra-foundations which I will do shortly )
[14:02:54] <btullis>	 elukey: That is quite a list! 
[14:03:16] <btullis>	 Thanks for going there.
[14:08:54] <btullis>	 Emperor:  Is it possible that this related to UEFI vs legacy booting? I see you did this https://github.com/wikimedia/operations-puppet/commit/5b96573f4ea35718078bf111551adf6bb7853426 so the hosts are *not* using UEFI, right? I got caught out recently by the fact that the `sre.hosts.provision` cookbook now uses UEFI by default and you have to pass `--legacy` if you want to use legacy BIOS booting.
[14:10:14] <Emperor>	 btullis: I didn't re-provision the host, so I don't think I'd have made any changes in that regard
[14:13:03] <elukey>	 btullis: yeah and we have two weeks to go through it, including the k8s services that are not mentioned
[14:13:38] <elukey>	 Emperor: never heard of the issue, but maybe the efi partition duplication logic could be at fault here, it is the only thing that I can think of from your description
[14:13:49] <elukey>	 are those hosts new or with a different config?
[14:14:57] <Emperor>	 elukey: ancient hosts config-J, I ran the convert-disks cookbook (which did a firmware upgrade by default to 7.0.0.183)
[14:15:14] <Emperor>	 I think it's still BIOS not UEFI
[14:17:35] <elukey>	 Emperor: ah ok, and is the idrac upgrade needed? I am wondering if downgrading it and the retrying would make a change
[14:18:09] <Emperor>	 elukey: the convert-disks cookbook does it (unless you disable). Is there a better version I should try?
[14:18:10] <elukey>	 or upgrading to another firmware version
[14:18:46] <Emperor>	 7.0.0.183 was the latest version offered, I think.
[14:19:04] <elukey>	 and do you recall the previous  version?
[14:19:05] <Emperor>	 elukey: I've created T423286
[14:19:06] <stashbot>	 T423286: Initial puppet run makes ms-be2068 unbootable - https://phabricator.wikimedia.org/T423286
[14:19:16] <Emperor>	 elukey: 5.something I think
[14:20:19] <Emperor>	 elukey: 2026-04-14 08:25:55,277 mvernon 3035179 [INFO] ms-be2068.codfw.wmnet (IDRAC): target_version: 7.0.0.183, current_version: 5.0.20.0
[14:20:30] <elukey>	 Emperor: I'd suggest to try to rollback to it on one host, then retry the reimage and see how it goes
[14:23:57] <Emperor>	 elukey: the cookbook isn't offering me 5.0.20.0, only 6.10.30.00 6.10.30.20 and the 7 one
[14:24:16] <Emperor>	 I'll try 6.10.30.20
[14:25:53] <elukey>	 the other alternative is to check with dcops if there is a newer firmware
[14:26:07] <elukey>	 at the moment the cookbook gets the firmware manually downloaded on cumin hosts
[14:26:23] <elukey>	 so the list is not updated unless dcops works on it
[14:28:09] <Emperor>	 I'll see how this goes (though I'm not sure what mechanism might explain the puppet-breaks-bootability being impacted by firmware)
[14:32:21] <Emperor>	 That might have broken it entirely :(
[14:33:20] <Emperor>	 yep, now console com2 says 111 (which I think is connection refused), and the cookbook's attempts to get a RedFish connection likewise are failing
[14:34:36] <Emperor>	 ah, no, perhaps it is now getting there.
[14:36:17] <elukey>	 you may need to re-run provisioning 
[14:36:37] <Emperor>	 firmware downgrade complete, retrying re-image
[14:51:56] <Emperor>	 post-installer reboot OK, now we wait for puppet to see if it breaks it again
[15:15:15] <Emperor>	 elukey: this time post-puppet it says "Booting from Hard drive C: GRUB" and hangs thus
[15:24:35] <elukey>	 Emperor: lovely, but it may give us a good indication that it is the firmware the issue
[15:26:02] <elukey>	 the sad part is that our 7.x idrac firmware seems to be the last one https://www.dell.com/support/product-details/it-it/product/poweredge-r740xd2/drivers
[15:26:55] <Emperor>	 elukey: I'm not entirely convinced - something (presumably puppet) has mucked up the GRUB install enough to render the system entirely unbootable (and at quite an early stage in the GRUB process)
[15:27:55] <elukey>	 Could be yeah. If you have a host with a similar config, you can try to go through the whole process without the firmware upgrade 
[15:27:59] <Emperor>	 (I _could_ try the third available firmware version, but that rather feels like we'll be back in the same spot in another 45 minutes)
[15:28:01] <elukey>	 and see if it leads to the same issue
[15:28:22] <elukey>	 one thing that I'd try is the idrac firmware and the BIOS one, if they are not coupled together
[15:28:47] <Emperor>	 I can do that (the disks cookbook has a skip-firmware option, though presumably a too-old bios not working properly is why it defaults to updating)
[15:29:13] <Emperor>	 elukey: the firmware that's getting changed is the idrac one (and I think only the idrac one)
[15:29:50] <Emperor>	 ...but I've spent all day on just this one host, so I'll not start the other possible-victim until tomorrow, I think, since otherwise it'll be in a weird state overnight
[15:31:03] <elukey>	 yeah I would try to upgrade the bios too, the supermicros ones for example couple it together
[15:31:23] <elukey>	 I think they are separate, but maybe somewhere in the changelog there is a dependency
[15:32:45] <Emperor>	 elukey: want to suggest a BIOS version?
[15:33:42] <Emperor>	 currently it's on 2.12.2. There look to be 2.7.5 2.5.4 2.6.5 2.24.0 2.17.1 and 2.15.1 available
[15:40:15] <elukey>	 most recent, 2.24.0 afaics (upstream provides 2.26 too)
[15:40:26] <Emperor>	 OK, doing that
[15:51:34] <Emperor>	 Done, host failed as before, but I'll try a re-image (seems unlikely to fix anything, but...)
[16:01:29] <elukey>	 lemme know, we can sync tomorrow in case
[16:01:36] <elukey>	 (need to go afk in a few)
[16:04:24] <Emperor>	 sure, installer running now, it'll be a while yet, and I don't think we're trying anything else today.
[16:07:43] <Emperor>	 post-installer boot OK
[16:33:22] <Emperor>	 post-puppet no boot, just "GRUB "
[17:01:14] <James_F>	 rzl: Progress on the AW/WF caching issues, the connections aren't connecting: T423311 Oops.
[17:01:16] <stashbot>	 T423311: Writes to /*/wf-wan/ failing with CONNECTION FAILURE or SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY (mcrouter not being reached?) - https://phabricator.wikimedia.org/T423311
[17:07:14] <rzl>	 James_F: I can take a look this afternoon!
[17:13:09] <James_F>	 Thanks!