[09:30:31] Can I get a +1 to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1272589 please? The easy part of the controller node rollover is the puppet change, the complicated bit is not stuffing up the key rollover... [09:42:04] sorry, in the middle of a maintenance [09:42:12] and I am in the middle of the incident [09:46:09] looking [10:13:23] TY :) [10:37:01] I updated https://wikitech.wikimedia.org/wiki/SRE/Data_Persistence/Documentation/Python with a bunch of suggestions if people are interested [10:40:43] Alas, I missed the need to also change roles around - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1272609 [puppet is disabled on apus-codfw in the mean time] [10:42:40] So I'd appreciate a review of that if anyone's got a moment; and at least in future we'll have documentation... [10:52:24] I gave it a look, and the patch is ok, but I didn't check that hiera changes were ok [10:53:52] thanks [11:16:23] done [11:18:42] cephadm key rollover complete. [12:04:44] I've drained the two outgoing backends, can I get a +1 to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1272676 to take them out of hiera so I can decommission them, please? [12:06:16] done, going for a covfefe [12:06:45] thanks, enjoy the coffee :) [13:26:13] I think my last CR for today is adding two storage nodes to eqiad apus - if I could get a +1 to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1272713 please? [13:26:29] [assuming all good over the weekend, I'll do the rollover & decom of the outgoing nodes on Monday] [15:27:26] docs on the key rollver process - https://wikitech.wikimedia.org/wiki/Ceph/Cephadm#Cephadm_key_rollover_/_Replacing_the_controller_node [16:57:47] @marostegui I suggest we create a dedicated task for https://phabricator.wikimedia.org/T327300#11818029 - right now we have that information in https://zarcillo.wikimedia.org/ui/hosts but to depool primary/dc masters we would also need automated switchover