[14:59:59] <inflatador>	 nice article about debugging CNI in a homelab environment: https://medium.com/@vikashkaushik232/my-weekend-kubernetes-project-turned-into-a-3-day-debugging-nightmare-dd56b16c87dd
[17:17:47] <elukey>	 Hey folks! Something nice found while debugging with Dawid failures creating pods on ml-serve1012 (trixie k8s 1.23)
[17:18:05] <elukey>	 plugin type="calico" failed (add): failed to find plugin "calico" in path [/usr/lib/cni]
[17:18:54] <elukey>	 elukey@ml-serve1012:~$ containerd config default | grep /usr/lib/cni
[17:18:54] <elukey>	       bin_dir = "/usr/lib/cni"
[17:18:56] <elukey>	 ta daaan
[17:19:00] * elukey cries in a corner
[17:19:44] <elukey>	 so in theory we should pass the right bin_dir down to the containerd's toml, but I didn't find a clean way without duplicating the info that we are using /opt
[17:19:54] <elukey>	 (from our calico configs)
[17:28:21] <elukey>	 I guess we could pull kubernetes::clusters_defaults in the containerd's profile, get the cni bin dir configured for the cluster and pass it down to container's class if we run trixie
[17:28:24] <elukey>	 thoughts?
[17:29:07] <jayme>	 elukey: so they changed the default in trixies containerd version? Or did I get that wrong?
[17:29:40] <elukey>	 jayme: exactly yes
[17:29:49] <jayme>	 the f...
[17:30:00] <elukey>	 I added it manually to ml-serve1012's config and it works nicely
[17:30:45] <elukey>	 now we are relying on the containerd's defaults to be the same as kubernetes::clusters_defaults, we could in theory simplify and just use the correct path on trixie
[17:30:49] <elukey>	 without pulling from hiera
[17:31:09] <elukey>	 but we may end up in some weird-to-debug use cases if we'll change it
[17:32:59] <jayme>	 IIRC the actual kubernetes config does not make use of the cni_bin_dir on k8s>1.23 so we could move it to the containerd profile
[17:36:47] <jayme>	 maybe the --cni-* arguments to kubelet can even be removed on k8s 1.23 when not using dockershim...I never tried that tbh
[17:38:47] <jayme>	 anyways...I would suggest to handle that in the containerd config template and just set /opt/cni/bin there - which makes it pretty explicit
[17:39:13] <elukey>	 ok so if trixie set that, without any hiera lookup?
[17:39:32] <jayme>	 I would set it unconditionally
[17:39:40] <elukey>	 +1 will send a patch
[17:43:07] <jayme>	 cool, thanks
[18:11:00] <elukey>	 filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1203500, will check it more in depth tomorrow