[10:24:42] <elukey>	 hey folks
[10:24:53] <elukey>	 as FYI I deployed spicerack 13.0.0 on cumin2003 as  test
[10:25:16] <jynus>	 are there important changes?
[10:25:26] <elukey>	 the major/visible change is that all the cookbooks that were requesting the pwstore's management password (reimage/provision/etc..) are now skipping the step
[10:25:36] <jynus>	 nice <3
[10:25:50] <elukey>	 because the password is stored in /etc/spicerack/management
[10:26:29] <elukey>	 should be one less hassle for people using cookbooks
[10:26:49] <elukey>	 once I tested everything I'll roll out the changes to the other cumin nodes as well
[10:30:19] <cezmunsta>	 *\o/* 
[10:31:27] <jynus>	 how is pwsecret and puppet synced at the moment, manually?
[10:33:22] <moritzm>	 if we rotate the management password, it will need to be update in pwstore and the private Puppet repo
[10:33:25] <elukey>	 yeah correct, not ideal but we rarely change the pwd
[10:33:42] <elukey>	 we have a cookbook coming up to rollout a new password
[10:34:23] <moritzm>	 until about two years I had the old management password memorised, that's how rare we change it :-)
[10:34:42] <marostegui>	 moritzm: it is not hard to memorize 12345
[10:35:29] <fabfur>	 I've got the same combination on my luggage
[10:35:57] <elukey>	 marostegui: don't cookbook bad feelings to people please
[10:36:43] <marostegui>	 hahaha
[10:36:55] <elukey>	 (that was good I know)
[10:37:08] <fabfur>	 going to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1305635
[10:37:11] <jynus>	 oh, I am not worried about current status (although I would love to have a proper service in the long term), I was asking in case it I had to update it
[10:37:15] <marostegui>	 elukey: not bad for you, not bad
[10:44:14] <moritzm>	 I'm going to fail over the URL downloaders to the new trixie VMs, this has to be rolled back last week, since squid got oom-killed due to increased RAM usage with Squid in trixie (and the specs were still from a time when the URL downloaders mostly proxied some Citoid, while they are being used a lot more with hcaptcha)
[10:44:18] <moritzm>	   the RAM has expanded from 2G to 8G and the cores also doubled,so that should no longer be an issue
[10:44:26] <moritzm>	 ^ arnaudb, XioNoX
[10:44:45] <moritzm>	 (for your remaining 20 minutes,after that I'm on-call anyway)
[11:31:31] <arnaudb>	 thanks for the heads up moritzm ! 
[14:33:56] <elukey>	 spicerack rolled out to all cumin nodes, no more pwstore management password 
[14:34:02] <elukey>	 if you see any issue please ping me
[14:35:00] <cdanis>	 amazing, thanks Luca
[14:35:02] <sukhe>	 woah nice!
[14:35:26] <fabfur>	 👍
[14:35:59] <jhathaway>	 nice work elukey!
[14:41:31] <godog>	 very cool
[15:02:10] <sukhe>	 taavi: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/0fc9cc1a187dd1fc7550b46103e09e23831ba7c2
[15:02:14] <sukhe>	 https://puppetboard.wikimedia.org/nodes?status=failed
[15:02:17] <sukhe>	 probably needs a look
[15:02:25] <sukhe>	 > Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/profile/monitoring/check_fresh_files_in_dir.py
[15:02:53] <taavi>	 looking, thanks
[15:03:01] <sukhe>	 <3
[15:05:36] <taavi>	 sukhe: spot-checked a few nodes and puppet runs successfully there on a retry
[15:06:22] <taavi>	 so I think those nodes started puppet runs timed just right so they got an old catalog still referring on the file but the file was gone by the time the agent got around requesting it
[15:06:32] <taavi>	 sorry for the noise!
[15:06:32] <sukhe>	 ah interesting. thanks!
[21:29:04] <inflatador>	 does anyone know where I can check quotas for thanos-swift S3 buckets? We have to restore a 2.2 TB opensearch index and I wanted to make sure we have enough space on our `elasticsearch-snapshot` bucket