[08:25:40] hey folks [08:26:02] very soon we'll need to rotate the PKI's discovery intermediate keypair for https://phabricator.wikimedia.org/T420993 [08:26:30] we are testing with the debmonitor's intermediate (expiring as well) a procedure in https://phabricator.wikimedia.org/T420993#11812923 for bare metal nodes [08:26:46] so ideally we'll have more experience before we attempt the discovery one [08:27:44] my main worry though is k8s - is there a way to precautionary stop cert-manager / cfss-issuer from renewing certs temporarily? It would be nice to have a sort-of puppet disable for prod clusters when we upgrade the discovery intermediate [08:28:08] I know it should handle everything transparently, but if something goes south we may end up in a big outage [08:28:11] thoughts? [08:44:48] I guess we could scale down the cert-manager replicas to 0 [08:48:23] yes, that probably works [08:51:56] brouberol: I don't see anything super obvious from the google doc [08:52:30] same [08:52:34] but it also states that a full admin_ng deploy was run which would have re-created those certs [09:08:20] yep, IIRC we did run `helmfile -e dse-k8s-eqiad -l name=cert-manager apply` [09:12:21] jayme: is it something that we can test somewhere? Maybe in staging, and see if certs are renewed [09:13:17] brouberol: that's different at it only applies the cert-manager release. The certs per namespace are created with the namespace release [09:13:32] elukey: yeah, you can do on staging-codfw [09:13:43] but we did run full admin_ng after that [09:14:06] that would have re-created the cert objects in namespaces ;) [09:14:19] yep, so I don't know what happened there tbh [17:08:00] anyone have a minute for a stamp on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1270494 [17:10:45] done [17:10:50] thanks!