[17:17:37] We discovered a problem when re-imaging k8s control planes: https://phabricator.wikimedia.org/T380142 [17:17:53] I would propose to refrain from doing so until that is fixed [17:18:48] jayme: ftr I'm +1 on your proposed quick fix [17:18:56] I'll post that on the task actually [17:19:15] sweet - I was about to ask if anybody has a better idea or objections [17:20:05] not in the short term -- I think fingerprint is sensible, and we can worry about auto-expiry/cleanup etc after un-blocking k8s control plane maintenance [17:22:35] I'll have to test if kube-apiserver is unhappy if a cert is provided twice - but I don't think that's the case. And auto-expiry/cleanup the way I proposed it might as well be good enough as kube-apiserver def. does not complain about expired certs in the list [17:23:53] the funny openssl thing I learned today is -noout does ofc. not noout when used with -checkend [17:24:03] do we need to have a revocation story 😅 [17:24:39] revocation is not a concept in k8s world in general 😇 [17:25:07] perfect [17:26:38] yeah...there is a note somewhere in the docs, like on the last lines about PKI and all those things. That reads something like 'don't use client certs for authentication (what this doc was talking about for 5 pages) as there is no way to revoke certs apart from switching to a different intermediate [17:27:40] so we should move to a OIDC provider for user authentication at some point...yay [17:27:46] not sure about machines, though [17:29:28] I mean, maybe? we aren't really authenticating users to the cluster, exactly [17:31:17] right. But we maybe should. As rn we hand out the keys to the deployer group [17:31:32] without a revocation story ;) [17:32:21] mmm yeah ok [17:35:55] maybe not directly relevant, but Openbao (OSI fork of Hashi Vault) is picking up steam. Never used it for k8s (we had nomad at the old job), but it does check the revocation box https://github.com/openbao/openbao [17:36:39] inflatador: jhathaway is looking at openbao for at least the Puppet secrets use case [17:37:16] yes, as well as k8s, though the initial target would be puppet [17:37:34] and k8s would consume secrets via its current method of puppet -> disk -> k8s [17:40:21] We used it for PKI and nomad secrets (kv engine, ref https://developer.hashicorp.com/vault/api-docs/secret/kv/kv-v2 ). I'm sure we barely scratched the surface of what it can do [19:44:58] does istio ingressgateway support generating http access logs? do we have an easy recipe for that in our production?