[13:10:11] taavi: I'm trying to upgrade the first toolsbeta control node and I'm seeing... [13:10:12] FATAL: failed to get node registration: node toolsbeta-test-k8s-control-10 doesn't have kubeadm.alpha.kubernetes.io/cri-socket annotation, or kubelet instance config /var/lib/kubelet/instance-config.yaml [13:10:45] which... I think something similar happened last time but don't remember the details. Do you recall how we got past it? [13:12:56] hmm [13:13:47] btw that is my second attempt; first attempt complained about that same .yaml file but iirc the message was slightly different. I can dig it up if it's useful [13:14:13] yes please [13:14:46] also, for what it's worth, instance-config.yaml is not present on -11 either [13:16:41] the annotation exists on toolsbeta-test-k8s-control-11/12 and all tools control nodes [13:16:54] we can patch the annotation in to -10 as well, but i am really curious how this got to this state [13:19:47] ok, I'm maybe wrong about the previous error message. Looking in the logs I don't see any real explanation: [13:19:54] https://www.irccloud.com/pastebin/ug9uDkqq/ [13:19:57] but still digging [13:20:08] So maybe the error about the missing .yaml file is just a result of me trying to re-run after a failure? [13:20:59] your logs show the upgrade going much faster than where it failed for me just now [13:21:42] going faster or failing faster? [13:21:49] further* [13:22:10] spicerack does not log command output, which is why the docs at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes#Upgrade_control_nodes say to save the command output [13:22:31] since right now your logs only show the upgrade dying midway during execution but show no more detail [13:25:16] yeah. I should've tmux'd -- my laptop decided to try to upgrade itself overnight and closed the terminals :( [13:26:12] please do not run any of the kubernetes upgrade cookbooks outside a tmux ever [13:26:46] but if the output is gone, i can try patching the annotation back in and we can try again and see how it goes this time [13:27:19] that's fair. The reimage cookbook enforces that, I almost think we should enforce that for /every/ cookbook. [13:27:44] send a patch? :P [13:28:04] andrewbogott: ok, annotation patched back in. you can try to run the cookbook again [13:29:10] running! [13:29:15] (in screen actually and not tmux because [13:29:18] I'm old) [13:30:23] "error: error execution phase kubelet-config: error reading kubeadm flags file: failed to read file: open /var/lib/kubelet/kubeadm-flags.env: no such file or direct [13:30:23] ory" [13:30:43] that's right after I approved the plan [13:31:05] https://www.irccloud.com/pastebin/RdnzOhZ7/ [13:32:14] hmm [13:32:29] do you remember if this was the thing it failed on the last time? [13:32:51] I think it was. [13:34:05] ok, hmm [13:34:28] there are some similar error reports but not exact... [13:36:12] that file seems to be missing entirely for all tools and toolsbeta k8s controls [13:36:30] yeah [13:36:33] if i look at an unrelated kubeadm cluster i have lying around: [13:36:33] taavi@k8scontrol08 ~ $ sudo cat /var/lib/kubelet/kubeadm-flags.env [13:36:33] KUBELET_KUBEADM_ARGS="--pod-infra-container-image=registry.k8s.io/pause:3.7" [13:37:38] Do we just have an edge case where there weren't any flags, and so the file didn't get written, and the new updated didn't account for that? [13:37:43] *updater [13:38:22] so looking at modules/kubeadm/templates/init.yaml.erb, we do override pod-infra-container-image in our environment [13:39:07] huh [13:39:22] btw, https://github.com/kubernetes/kubeadm/issues/3279 is a similar thing but I think that's when the file exists but is empty. [13:39:55] but it seems like that for whatever reason, kubeadm hasn't written the file in our cluster [13:39:56] the other possible edge case is: we are running the oldest in-place-upgraded cluster in the world :) [13:40:39] gemini tells me "kubeadm started writing /var/lib/kubelet/kubeadm-flags.env in Kubernetes v1.11" [13:40:44] also, oops https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/049bafcc8cc7bdb20828ab1753111dcaa1148c02%5E%21/#F0 [13:40:54] (that setting only affects bootstrapping new clusters, but still :P) [13:42:06] yeah! That wasn't going to work [13:42:43] Our clusters were born before 1.11 weren't they? So maybe we should just manually drop in that file :/ [13:42:45] nope [13:42:58] nope, don't predate 1.11? [13:43:00] at this point i'd try creating them empty (like https://github.com/kubernetes/kubeadm/issues/3279#issuecomment-3724154135) and seeing what happens to them during the upgrade [13:43:14] ours was initially 1.16 plus minus a couple iirc [13:43:19] ah, ok [13:43:20] so not pre-1.11 [13:43:34] so empty, not with pod-infra-container-image ? [13:43:38] empty [13:43:55] ok. And, tell me how to roll back to before the failed upgrade? [13:44:25] you just run the upgrade cookbook again when you're ready [13:44:45] ok... [13:44:46] andrew@toolsbeta-test-k8s-control-10:~$ sudo touch /var/lib/kubelet/kubeadm-flags.env [13:44:50] and now re-running [13:44:55] ... [13:45:48] i was trying to say empty as in the variable exists but is empty, like the comment i linked has? [13:46:21] oh! empty vs empty [13:46:47] sorry, should have been more clear [13:46:51] anyway, did that work? [13:47:18] no, but it threw the second round error, "doesn't have kubeadm.alpha.kubernetes.io/cri-socket annotat [13:47:18] ion, or kubelet instance config /var/lib/kubelet/instance-config.yaml" [13:47:57] ok, that's the thing we had earlier [13:48:06] * taavi fixes the annotations again [13:48:17] https://www.irccloud.com/pastebin/1S3R5mp1/ [13:48:28] try again [13:49:09] can you paste the 'fix the annotations' steps so I can add to the docs? Seems like that happens now and then. [13:49:40] i don't think that is needed unless the upgrade dies in-flight, so not a normal procedure [13:50:01] not normal but... has happened to me in this upgrade and the last [13:50:12] Also, argh! [13:50:19] error: error execution phase kubelet-config: error reading kubeadm flags file: failed to read file: open /var/lib/kubelet/kubeadm-flags.env: no such file or directory [13:50:24] And, indeed, the file is now gone [13:51:17] can i have the full output for the upgrade cookbook this time? [13:52:28] yep. [13:54:30] https://www.irccloud.com/pastebin/KxRqSQXm/ [13:55:05] Notice: /Stage[main]/Profile::Wmcs::Kubeadm::Core/File[/var/lib/kubelet/kubeadm-flags.env]/ensure: removed (corrective) [13:55:10] HMM [13:55:12] why, puppet? [13:55:35] T370245 [13:55:36] T370245: [infra,k8s] remove deprecated kubelet flags before 1.28 upgrade (we might be able to remove all custom ones) - https://phabricator.wikimedia.org/T370245 [13:56:02] ok let's revert that! [13:56:07] its done its job [13:56:13] yes [13:56:39] also this explains approximately everything [13:56:48] other than the classic 'how did this ever work' [13:57:47] one minute while I tune up this patch... [13:58:01] good find [14:00:41] extremely obvious in retrospect! [14:00:42] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1372707 [14:01:20] i wouldn't be surprised if this also caused some of the problems last time [14:01:24] +1 ship it [14:01:27] Likely! [14:01:52] while waiting for CI... going to just populate that file on all the controllers right now [14:02:28] you'll need it on all the workers as ewll [14:03:34] oh... ok then it's cumin time [14:05:51] taavi: I'm not here, but let's add wmflib.interactive.ensure_shell_is_durable if it should never be run not in tmux/screen ;) [14:06:25] volans|off: if you ask me that should be in all cookbooks, period [14:06:45] :D [14:07:09] agreed! [14:07:40] huh, here's a silly cumin question, what do I do about the third nested quotes in [14:07:41] 'echo 'KUBELET_KUBEADM_ARGS=""' > /var/lib/kubelet/kubeadm-flags.env' [14:07:53] can't believe I haven't faced this before [14:08:53] my lazy solution would be 'echo ... | base64 -d > ...' [14:09:08] that or escape [14:09:25] AI are great at this kind of silly things ;) [14:22:06] ok, that's all the backfilling done, taavi are we ready to re-run the upgrade cookbook yet again? [14:24:00] andrewbogott: go for it [14:25:17] ok, at least puppet didn't rm the file this time... [14:25:53] doesn't like the empty file though [14:26:27] error reading kubeadm flags file: no flags found in file "/var/lib/kubelet/kubeadm-flags.env" [14:26:50] which, for completeness, is [14:26:53] https://www.irccloud.com/pastebin/4lSjQJGY/ [14:27:26] bleh [14:27:33] that's the bug we were looking at earlier? [14:27:56] yeah, although the fix suggested there is what we're doing... [14:28:16] "Now, the env file can legitimately be empty, but ReadKubeletDynamicEnvFile() in cmd/kubeadm/app/phases/kubelet/flags.go treats an empty flags list as an error." [14:28:39] but we've already tried legitimately empty haven't we, with 'touch'? [14:28:40] no, that comment is not a suggested fix [14:29:19] the fix is either in k8s 1.35 or 1.36 (can't read from that github thing), and we're upgrading to 1.34 [14:29:57] i think we just need to add some flag in there, like the pod-infra-container-image one [14:30:06] KUBELET_KUBEADM_ARGS="--this-is-fake=fake" [14:30:14] that'd be --pod-infra-container-image=docker-registry.svc.toolforge.org/pause:3.9 these days [14:30:22] i bet that'll fail as an unknown flag [14:30:47] yeah [14:30:53] considering how carefully validated this is [14:32:13] https://www.irccloud.com/pastebin/MoP0jWqd/ [14:33:10] lgtm [14:33:41] are we reset and ready to try again? [14:34:22] sure [14:38:31] It's still unhappy but I think unimportantly... [14:38:39] https://www.irccloud.com/pastebin/i5Hb0Fvq/ [14:39:08] that's just leftovers from previous failure I think? [14:39:15] that's just leftovers of a previous failed upgrade i think [14:39:16] .. yes [14:39:28] And since it got to uncordoning, we're good right? I'll just rm that file [14:39:30] or lemme check [14:39:54] yeah, rming that file is fine [14:40:24] ok. Going to move on to the other nodes now -- with luck this will be the last you hear from me :) [14:40:27] WAIT [14:40:30] waiting :) [14:40:49] so the thing that failed is the 'fix' for https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes#Permission_errors_after_control_plane_upgrades, so please double-check extra carefully that the components came up successfully on the just upgraded node before moving on [14:41:03] if not, there's a cookbook to do that step only that you'll need to run [14:41:20] by 'components' i mean kube-apiserver / -scheduler / -controller-manager [14:43:47] I see three of each of those running in 'get pods --all-namespaces' and I see each of them listed in 'describe node' under 'Non-terminated Pods' [14:44:13] that should convince me that all is good, yes? [14:44:58] no, you need to look at the pod logs in detail [14:46:05] and make sure they don't have the lease error I linked above [14:48:28] the apiserver log is full of errors but not /that/ error. [14:48:33] https://www.irccloud.com/pastebin/duFaEdvz/ [14:49:13] oh, but controller-manager does have that [14:49:54] that's, uh, probably fine during the upgrade [14:50:06] as long as you don't have /that/ error, you're fine to continue imo [14:50:16] controller-manager has [14:50:16] error retrieving resource lock kube-system/kube-controller-manager: leases.coordination.k8s.io "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system" [14:50:21] ok, so that is that error [14:50:30] So we need to reset. What cookbook are you referring to? [14:50:36] wmcs.toolforge.k8s.restart_static_pods [14:55:06] ok, logs seem happier now apart from all the 'Too large resource version' [14:55:43] that's fine during an upgrade [14:55:53] yep [14:56:04] so... anything else to check before -11 ? [14:56:23] I guess I can run the functional tests [14:56:34] imo you're good to continue, although i'd double check if you have the same leftovers (/etc/kubernetes/manifests/.cookbook-stopped-*) on the other nodes [14:56:41] i doubt the tests are that useful mid-upgrade [14:56:54] ok [15:03:04] -11 upgraded fine [17:42:09] did someone from our team get hailed on the etherpad issue? there was a question (much earlier today) about needing to change a password for wmcloud one [17:44:55] I think the only etherpad implementation on cloud-vps was 'etherpad-bookworm' which was shut down today, "Nov. 20, 2026, 4:56 p.m." [17:45:20] that VM was created by SRE Daniel Zahn so I imagine it's up to him or his cohort to fix or delete. [17:45:46] (Oh, to answer your actual question: I wasn't pinged about it, I don't know who shut it down) [17:46:08] oh, wait, I do, it's in the log! lsobanski shut it down. So likely no one in our team was involved. [17:57:21] yes it was clear from the log that it was shutdown by those who responded to the concern, but there was a question of who is changing credentials (in another IRC channel) and I was looking for information on whether that should be someone from our team. and you answered that question (it should be Daniel). thank you!