[08:58:30] <godog>	 for the folks around today: I'm seeking a quick review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1120130
[09:06:39] <godog>	 thank you arnaudb !
[09:07:38] <arnaudb>	 np!
[14:45:11] <elukey>	 I am working on kartotherian, surely my fault
[14:45:20] <elukey>	 checking what's wrong, I just pooled more k8s nodes
[14:45:41] <volans>	 ack
[14:46:02] <elukey>	 !incidents
[14:46:02] <sirenbot>	 5681 (UNACKED)  [3x] ATSBackendErrorsHigh cache_upload sre (kartotherian.discovery.wmnet)
[14:46:12] <elukey>	 !ack 5681
[14:46:12] <sirenbot>	 5681 (ACKED)  [3x] ATSBackendErrorsHigh cache_upload sre (kartotherian.discovery.wmnet)
[17:09:44] <elukey>	 This is the summary of the incident for whoever is interested: https://phabricator.wikimedia.org/T386648
[17:09:51] <elukey>	 I don't have a clear root cause yet
[20:08:09] <Krinkle>	 Has something changed about how to apply a puppet patch in beta cluster? I've got it cherry picked in `root@deployment-puppetserver-1:/srv/git/operations/puppet` but it seems no matter what I do, running `sudo run-puppet-agent` on deployment-mediawiki14 still runs with the previous commit.
[20:08:19] <Krinkle>	 Info: Caching catalog for deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud
[20:08:20] <Krinkle>	 Info: Applying configuration version '(67c7ef4cd5) gitpuppet - deployment-prep: Remove parsoid things from hiera'
[20:08:34] <Krinkle>	 root@deployment-puppetserver-1:/srv/git/operations/puppet# git log --oneline -n2
[20:08:35] <Krinkle>	 4f069b799e (HEAD -> production) mediawiki: Add rewrite rule to fix serving of /.well-known static files
[20:08:35] <Krinkle>	 67c7ef4cd5 (tag: snapshot-202502171709) deployment-prep: Remove parsoid things from hiera
[20:41:33] <Krinkle>	 Well, time passing doesn't make it fix itself. Still out of sync 40min later.
[20:41:53] <Krinkle>	 I checked /etc/puppet/puppet.conf does point to puppetserver-1
[20:42:52] <Krinkle>	 on deployment-puppetserver-1:/var/log/syslog it reports various other servers running puppet are also using the old commit still at the current minute, so it is getting it from somewhere on that server.
[20:54:56] <Krinkle>	 The most common reason that a patch fails is if it causes some kind of compiler error, in which case it would report that on puppet agent, it wouldn't keep provisioning the previous commit.