[08:28:25] <wm-bb>	 <lucaswerkmeister> I’m getting lots of emails about failed puppet runs on Cloud VPS machines
[08:28:27] <wm-bb>	 <lucaswerkmeister> ERR: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node special-new-lexeme-testing.wikidata-dev.eqiad1.wikimedia.cloud: Exception while executing '/usr/local/bin/puppet-enc': Cannot run program "/usr/local/bin/puppet-enc" (in directory "."): error=0, Failed to exec spawn helper: pid: 868905, exit
[08:28:27] <wm-bb>	 value: 1
[08:28:45] <wm-bb>	 <lucaswerkmeister> known issue? I don’t see anything in #wikimedia-operations
[08:34:20] <arturo>	 I'm investigating
[09:03:00] <wm-bb>	 <Vincent> I'm spammed by puppet failures as well
[09:04:58] <arturo>	 !log cloudinfra restart puppetserver service in T377803
[09:05:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[09:05:01] <stashbot>	 T377803: Cloud VPS:  cloud-wide puppet problem related to puppet-enc 2024-10-22 - https://phabricator.wikimedia.org/T377803
[09:05:53] <arturo>	 !log tools restart puppetserver service for T377803
[09:05:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:12:54] <wm-bb>	 <Vincent> Thank you Arturo :)
[09:15:52] <arturo>	 you are welcome
[09:15:58] <wm-bb>	 <lucaswerkmeister> thanks \o/
[09:16:20] <arturo>	 the phab ticket contains a bit more info about what happened
[13:23:37] <inflatador>	 hooray for puppet ;(
[13:25:04] <inflatador>	 Had a question about this ticket ( https://phabricator.wikimedia.org/T377655#10249075 ) ...do we normally allow mounting the dumps server from WMCS hosts?
[13:30:14] <inflatador>	 looks like yes https://wikitech.wikimedia.org/wiki/Help:Shared_storage
[14:49:32] <taavi>	 I'm still getting Puppet failure emails from tools :/
[14:52:48] <arturo>	 taavi: yeah, that one is on me
[14:52:50] <arturo>	 https://www.irccloud.com/pastebin/crB5jxG6/
[14:53:31] <taavi>	 arturo: no, just our exim throttling on the mail relay and the mail queue on the individual nodes working as (individually) expected
[14:53:51] <arturo>	 oh, ok
[15:02:15] <taavi>	 !log admin recover access to User:Labslogbot T376220
[15:02:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:02:21] <stashbot>	 T376220: Labslogbot needs new SUL OAuth credentials after Wikitech authn changes - https://phabricator.wikimedia.org/T376220
[18:59:11] <Guest21>	 X-P X) (y) O:3