[09:14:12] <dcaro>	 !log tools migrating pipelineruns stored versions to v1 (T376710)
[09:14:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:14:17] <stashbot>	 T376710: [builds-builder,builds-api] After the upgrade build actions take >10s - https://phabricator.wikimedia.org/T376710
[11:58:00] <jnuche>	 👋 heya, puppet has been failing for roughly a month on one of our cloud vps hosts: k3s.catalyst.eqiad1.wikimedia.cloud
[11:58:06] <jnuche>	 ERR: Could not retrieve catalog from remote server: Request to [...] failed after 0.012 seconds: SSL_read: sslv3 alert certificate unknown
[11:58:06] <jnuche>	 Wrapped exception:
[11:58:06] <jnuche>	 SSL_read: sslv3 alert certificate unknown
[11:58:06] <jnuche>	 WARNING: Not using cache on failed catalog
[11:58:07] <jnuche>	 ERR: Could not retrieve catalog; skipping run
[11:58:26] <jnuche>	 it reproduces when running the agent manually on the machine. Seems something happened to the agent certificate?
[11:58:55] <jnuche>	 wondering if a kind soul here could help with this
[12:23:59] <dcaro>	 jnuche: let me try refreshing the cert
[12:44:36] <jnuche>	 dcaro: any luck?
[12:45:34] <dcaro>	 not really, it still fails, looking
[12:45:48] <dcaro>	 not sure where it's getting the certificate it uses :/
[12:46:13] <dcaro>	 does it have any special ssl config of any kind?
[12:47:32] <jnuche>	 none that I know of
[12:47:35] <jnuche>	 it should be all defaults
[12:48:22] <jnuche>	 I did notice the failures started after we did some maintenance on the instance back in Sept
[12:48:33] <jnuche>	 the machine was put in rescue mode for a bit
[13:05:10] <dcaro>	 jnuche: done
[13:05:25] <dcaro>	 jnuche: I manually deleted the old certs and just ran puppet and it was able to refresh correctl
[13:05:28] <dcaro>	 *correctly
[13:06:34] <jnuche>	 dcaro: awesome, thanks so much!
[13:06:51] <jnuche>	 sounds like you did all of that on the instance itself? no need to touch the puppet master?
[13:34:09] <dcaro>	 yep correct :)
[13:47:21] <jnuche>	 thx again!
[14:11:31] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.sal webservice restart # responded with 404 Not Found from fourohfour tool o_O
[14:11:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL