[07:43:53] <godog>	 greetings
[07:51:04] <dhinus>	 morning!
[07:51:18] <dcaro>	 morning
[07:55:24] <taavi>	 o/
[08:40:20] <godog>	 FYI I'm going to test a reboot of the new nfs server in toolsbeta toolsbeta-nfs-4
[09:41:31] <godog>	 went better than expected, I'll fix T347681 so we don't need a puppet run on nfs server reboot
[09:41:32] <stashbot>	 T347681: Cloud VPS: NFS servers: the current setup requires a puppet run after a reboot to get address right - https://phabricator.wikimedia.org/T347681
[09:43:00] <dcaro>	 \o/
[09:44:33] <dcaro>	 hmm, tools does not have the loki storage secrets 
[09:48:00] <dcaro>	 restored the lost commit
[09:49:26] <taavi>	 wdym by "does not have secrets"?
[09:55:51] <dcaro>	  /etc/toolforge/secrets.yaml file was missing those secrets, as the commit was removed from the git history in puppetserver (mistakenly), sorted
[09:56:02] <taavi>	 uhhh, how did that happen?
[09:56:18] <dcaro>	 manuall error resetting the private repo to main
[09:56:21] <dcaro>	 *manual
[10:50:12] * dcaro lunch
[12:50:15] <taavi>	 looking for reviews for tofu patches about tools-k8s-haproxy: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83
[13:15:35] <taavi>	 You should run a new pipeline, because the target branch has changed for this merge request. AAAAAAAA
[13:16:05] <taavi>	 one of these days I'll write something to automate that
[13:44:15] <godog>	 heheh
[15:32:02] <taavi>	 for dcaro's suggestion: https://bash.toolforge.org/quip/AU7VV2qB6snAnmqnK_0y
[15:32:25] <dcaro>	 lol
[15:42:52] <taavi>	 thoughts about T405584? (technically that and the immediately preceding request go over the 300G threshold of https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Clinic_duties#Community_Requests so needs one extra approval before proceeding)
[15:42:53] <stashbot>	 T405584: Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584
[15:52:12] <dcaro>	 +1d
[15:52:42] <dhinus>	 +1d
[15:54:45] <taavi>	 ty
[16:47:24] <dcaro>	 paws is down, let's take a look at the workers
[16:50:22] <dcaro>	 hmpf... ssh seems to hang :/
[16:51:29] <dcaro>	 I get a login prompt from the console, but I don't think it has a user/pass
[17:01:26] <dcaro>	 ssh was timing out, rebooted it, journal --boot -1 does not show anything interesting on a first glance
[17:32:09] * dcaro off
[17:32:11] <dcaro>	 cya on monday!