[09:00:37] <stw>	 I'm getting puppet failure emails on all my instances in account-creation-assistance. It looks like it's trying to run /usr/local/bin/puppet-enc, which doesn't exist
[09:02:02] <stw>	 Actually, if I read the error message correctly, it's an error 500 on server, with the server complaining about /usr/local/bin/puppet-enc
[09:04:50] <stw>	 Error 500 on SERVER: Server Error: Failed when searching for node [snip]: Exception while executing '/usr/local/bin/puppet-enc': Cannot run program "/usr/local/bin/puppet-enc" (in directory "."): error=0, Failed to exec spawn helper: pid: 1365284, exit value: 1
[09:07:25] <arturo>	 stw: I can reproduce, I will work on that
[09:14:22] <codders>	 (seeing the same on our instances special-new-lexeme-testing and wikidata-icinga-2024) 
[09:22:41] <arturo>	 !log admin fleet-wide restart of puppetservers T385553
[09:22:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[09:22:47] <stashbot>	 T385553: Cloud VPS puppet breakage on 2025-02-04 related to puppet-enc - https://phabricator.wikimedia.org/T385553
[12:48:44] <arturo>	 !log admin replacing cloudgw1002 with cloudgw1004 - T382356
[12:48:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[12:48:50] <stashbot>	 T382356: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356
[12:59:41] <Reedy>	 !log jouncebot restarting as it has dropped off IRC
[14:02:25] <Reedy>	 !log jouncebot restarting as it has dropped off IRC (second time)
[14:06:51] <Lucas_WMDE>	 Reedy: looks like stashbot is also having issues so I don’t think those !log’s went anywhere :S
[14:08:31] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.stashbot bin/stashbot.sh restart
[14:08:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[14:09:13] <Lucas_WMDE>	 that’s better
[14:10:07] <andrewbogott>	 thx Lucas_WMDE 
[14:31:47] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.stashbot bin/stashbot.sh restart
[14:31:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[14:50:23] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.jouncebot ./jouncebot/bin/jouncebot.sh restart # left #wikimedia-operations 14:46 UTC with “connection reset by peer”, nothing in kubectl logs
[14:50:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL
[14:50:53] <wm-bb>	 <lucaswerkmeister> feels like something is up with the network today :/
[15:03:11] <arturo>	 @lucaswerkmeister we had a network maintenance operation 1h ago
[15:03:55] <wm-bb>	 <lucaswerkmeister> ok, then I hope it’ll work better now
[15:42:29] <wm-bot>	 !log melos@tools-bastion-13 tools.stewardbots SULWatcher/manage.sh restart # SULWatchers disconnected
[15:42:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[18:15:31] <physikerwelt>	 some positive news, in the context of the reboot action of instances I discoverd that one can avoid problems with too large log files from docker by adding a single line to the puppet configuration https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ae6b6ce2349c5193a6f52b0ef69d9084ee664c67%5E%21/math/math-docker-2.math.eqiad1.wikimedia.cloud.yaml
[18:15:58] <physikerwelt>	 (maybe this is helpful for others)
[18:56:37] <subbu>	 andrewbogott, looks like I signed off before I saw your message which answers the qn. I came to ask here ... you want me to create a phab task to revert the temporary quota bump then?
[21:44:51] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed 2ccb28ad17 (l10n updates: lb)
[21:44:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL