[10:35:08] <stevemunene>	 Hi folks, facing an etcd reimage and upgrade issue I would like some help on. We've followed https://wikitech.wikimedia.org/wiki/Etcd#Reimage_nodes_a_cluster the first option and the reimaged bookworm verion was upgraded to 3.4.23 while the leader remains on the previous v3.3.25
[10:35:17] <stevemunene>	 while the cluster is healthy, there have been times we keep getting 404s for simple queries on the leader and certain responses which seem like they are voting
[10:36:03] <stevemunene>	 raft index, raft applied index are all good. should we choose a new leader and proceed with the reimage?
[10:42:34] <stevemunene>	 cc jayme 
[10:48:24] <jayme>	 hmm...I recall some shenanigans with mixed versions. Let me try to find details
[10:48:46] <jayme>	 did you check the upgrading notes from upstream? stevemunene
[10:53:50] <jayme>	 ah, no that was that etcd 3.4 (compared to 3.3) flips client-cert-auth to true (even if it is explicitely set to false) when trusted-ca-file is set - and they dont mention that in docs
[10:56:22] <jayme>	 stevemunene: I'd move the leader role to the newest node (simply so that it does not have to change again) and continue with the process. Although the impact is not super clear to me ("there have been times we keep getting 404s")
[11:06:48] <stevemunene>	 Thanks, I will be going through the docs ince more and our config just incase theres something that missed but all 3.4.23 hosts seem healthy so far
[14:29:06] <herron>	 on a similar topic I ran into etcd 3.4 building aux-k8s codfw, and have some open thoughts/questions about how to proceed wrt etcd backups/snapshots (because as-is the current backup script breaks under 3.4) in https://phabricator.wikimedia.org/T385727 and related patches.  thoughts and feedback there would be much appreciated
[15:54:13] <stevemunene>	 I think you would need to add the 'etcd --enable-v2=true' then use 'ETCDCTL_API=2'
[15:54:46] <stevemunene>	 The major breaking changes were the two on the v2 api
[16:01:42] <jayme>	 I woul assume that a backup with ETCDCTL_API=2 does backup the v2 database, not the v3 one
[16:20:40] <stevemunene>	 Oh yeah, so I would assume the option now would be as herron said on somehow having both untill all hosts are on v3 https://phabricator.wikimedia.org/T385727#10535814