[09:18:54] Hello. Could someone help explain why we have these KubeletOperationalLatency errors on the dse-k8s-eqiad cluster, please? https://alerts.wikimedia.org/?q=prometheus%3Dk8s-dse&q=alertname%3DKubeletOperationalLatency [09:20:18] They all seem to be affecting the `run_podsandbox` and `stop_podsandbox` operations. The behaviour matches what we see here, but I'm not sure how to alleviate it: https://grafana.wikimedia.org/goto/TydHTfYNR?orgId=1 [09:34:11] * akosiaris looking [09:35:19] I see that other clusters also have higher values for these ops than for others, but presumably not high enough to trigger the alert. e.g. https://grafana.wikimedia.org/goto/hpyZ0fLNR?orgId=1 [09:36:34] yeah, I was looking at that. codfw doesn't btw. But also codfw isn't running the periodic jobs [09:38:00] yeah, if you look at the codfw wikikube cluster there are very few of these events [09:38:55] now, the sandbox is an abstract concept in kubernetes. In the case of docker/containerd it's all about setting up/tearing down the namespaces when starting up a new pod (via the pause container) [09:39:17] oh and cgroups ofc too [09:39:55] I guess that in dse-k8s there's a lot of churn regarding pods being started often? Jobs are probably the typical example. [09:41:02] I don't think there's much we can do about this tbh. The counter productive solution would be to try and lower the rate of pod startup/stopping. Not much to gain there. [09:42:16] btullis: you can increase the threshold for this operations. There is an example in the alerts repository for list_images operation [09:43:22] you should be able to adapt it. But it looks like p99 reaches the 1.0 threshold at times, which is why it triggers [09:44:14] not sure what would be a good number tbh. It depends I guess on how fine you are with pods delaying a couple of seconds to start-up ? [09:44:30] akosiaris: OK, great. Yes, I think that increasing the threshold would be right for us. There is a lot of churn now, as a lot of our workload is coming from Airflow pipelines, which create a lot of ephemeral pods. [09:44:53] ah, there is a way to lower that metric. Moar hosts! [09:45:18] keep that in mind for the future, but for now, yes just bump it. [09:45:47] Cool, we're adding two new hosts today, and we have some more in the pipeline. Many thanks. [11:04:03] Does anyone have experience with PriorityClasses? We're trying to make sure that the core airflow components (webserver, scheculer, etc) get a higher priority class than the airflow task pods that get scheduled on the k8s cluster [11:04:19] well, s/trying/aiming at/ [11:32:53] Aside from calico and cert-manager, I don't think we 've used this feature yet in the WMF. [11:52:51] I saw that they were using the system-cluster-critical class. Is it something that we define somewhere, or does it come builtin? [11:52:55] (thanks btw( [11:56:07] brouberol: that's built in [11:56:11] https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/ [11:57:07] gotcha, thanks. So if we wanted to use a custom priority class, we'd probably need to fiddle with admin_ng to install them in the first place, as they are non-namespaced [12:23:46] yeah. You'll have to add the custom priorityclass as well as resourcequota objects. The latter are required to grant a namespace quora for a certain priorityclass [12:24:46] oh, interesting, thanks! I'll have a look at that as well, thanks [13:00:01] jayme: could I ask for a review of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1156319 please? [13:05:23] sure