[03:02:07] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[07:02:07] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ms-be1058:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[08:21:33] <Emperor>	 that's an expired silence
[08:28:41] <Emperor>	 (though we might have actually got to the point where it can be removed from the rings)
[09:38:24] <marostegui>	 btullis: when could I upgrade an-redactteddb1001?
[09:40:06] <btullis>	 marostegui: Thanks. Any time this week, would be fine. 
[09:40:15] <marostegui>	 btullis: I will do it now then!
[09:47:36] <marostegui>	 btullis: done, sections coming back up
[09:48:01] <btullis>	 Excellent! Cheers.
[10:29:40] <marostegui>	 jynus: to double check https://phabricator.wikimedia.org/T376905 the backups sources involved are up-to-date there?
[10:34:48] <jynus>	 I will do soon a check, as I have to check I have not missed any upgrades
[10:34:58] <jynus>	 or that any upgraded host has the latest stuff
[10:35:01] <marostegui>	 Thank you .)
[10:35:02] <marostegui>	 :)
[10:35:03] <jynus>	 or any other mistake
[10:35:12] <jynus>	 I was going to do it know that it is technically done
[10:35:24] <jynus>	 as a heads up, I also need to do a grant update in general
[10:35:45] <jynus>	 and that may be a lots of gradual deploys, including some prouduction, non-mw hosts
[10:36:01] <jynus>	 (just grants on non-primaries)
[10:36:53] <jynus>	 So T383902 may be up for some time even if all hosts are marked as done
[10:36:53] <stashbot>	 T383902: Upgrade backup source or mediabackup database host os to Debian bookworm or decommission them - https://phabricator.wikimedia.org/T383902
[10:37:09] <jynus>	 s/up/unresolved/
[10:38:03] <jynus>	 I also need to test the backups for upgraded hosts to confirm they work as intended
[11:20:01] <Emperor>	 Could I get a +1 to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114971 please? Remove some drained nodes from the swift rings so they can get decommd
[11:22:23] <marostegui>	 Emperor: check -sre, I cannot +1
[11:28:05] <Emperor>	 jynus: thanks, but you added V+1 :)
[11:28:29] <jynus>	 I see
[11:31:30] <marostegui>	 jynus: check -sre
[11:39:36] <Emperor>	 my CR now has the right sort of +1 available to you :)
[12:05:15] <jynus>	 If I can get a regex sanity check, too: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114986
[13:02:37] <Emperor>	 +1
[13:03:28] <jynus>	 thanks
[14:44:35] <Emperor>	 Can I get a +1 for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1115038 please? Nodes are now out of the ring, need removing from storagehosts before decomm.
[14:45:39] <Emperor>	 TY :)
[17:00:26] <marostegui>	 dhinus: Are you still aiming for clouddb* upgrades this week?
[17:01:27] <dhinus>	 yep, at least some... unless you have reasons for doing them later
[17:01:44] <dhinus>	 or for doing them urgently :)
[17:02:06] <marostegui>	 no no, this week is great
[17:02:30] <dhinus>	 okok, it's possible some will slip to next week, but I'll at least start with a few ones
[17:02:46] <marostegui>	 dhinus: Sure, no problem. Thanks
[17:03:05] <marostegui>	 If you can do at least the ones serving s3/s5 first, that'd be great
[17:17:18] <dhinus>	 marostegui: ok I'll start from those
[17:18:47] <marostegui>	 thank you
[18:13:52] <marostegui>	 jynus: All the backup sources that get upgraded to 10.6.20 get their tables rebuild right?
[18:13:59] <marostegui>	 Just to confirm
[18:14:55] <jynus>	 yes
[18:15:12] <marostegui>	 thank you!