[10:28:26] o/ [10:28:57] filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1270873 to verify what we discussed yesterday about cert-manager and the pki intermediate cert change [10:29:21] lemme know if it is too much or ok, I'll try to test it on staging today/tomorrow and then create a plan for prod [10:48:48] elukey: wouldn't it be enough to do this on one staging cluster rather then all of them? [10:51:08] also: it should be good enough to just set replicaCount: 0, cainjector and webhook rely on that (controller) cfssl-issuer can be kept running since it does not do anything on it's own [12:24:18] jayme: I'd prefer all of them to be sure, or at least one wikikube and one ML. Having both wikikube staging was nice so I didn't need to figure out the most used one, or see if one has more use cases then the other. It is just a quick deployment, in theory it shouldn't be an issue.. wdyt? [12:24:56] for the instances, got it, I wasn't sure if there was a chain of event that could have triggered new data to be injected without cert-manager pods [12:25:32] well...not refreshing staging-codfw certs is not an issue. not refreshing staging-eqiad will cause alerts to fire and things to break - that's why I'm asking [12:25:58] AIUI you wanted to verify that refreshes do no longer happen if cert-manager is scaled to 0 [12:26:20] that bahaviour will be the same regardless of which cluster [12:28:02] yep but I'd really like to confirm on more than one place that this is the case, with and without things that may break [12:28:13] what can happen in staging though? [12:28:28] I mean, I can understand endpoints may have stale certs [12:28:45] but if we announce it, it shouldn't be a a big issue, and I am not planning to do it for an extended time [12:29:38] not sure what can happen...just saying it might alert and break things compared to staging-codfw [12:30:42] but if you feel more safe shutting it down on all the clusters it's fine by me. I just wanted to say I don't think its necessary [12:31:15] yeah I know but I would really be happier to test on both :) [12:31:21] please bear with me [12:31:26] sure :) [12:33:59] <3 [12:34:15] ok I'll fix the replica count thing, and get back to you [13:14:49] aaand both wikikube staging clusters without cert-manager [13:26:42] Hello. I'm starting the work on the ipip migration of the dse-k8s clusters. Here's the first patch for the control plane servers: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1270929 [22:44:33] [non-urgent] I'd imagine this is something we've thought about, but I'm struggling to find prior art: have we looked into automating external-services network policy updates? (e.g., at least purely additive ones) [22:44:33] if we *have* talked about it, I figured this is probably the venue where someone will remember where :)