[06:44:01] * dcaro online [06:44:08] looking at the backlog [06:47:43] morning :) [06:51:37] morning ☕ [07:07:19] functional tests are not passing on tools, looking (read timed out when doing `build quota`) [07:09:08] two failures in a row [07:09:17] I'll open a task [07:41:59] those are dns issues, so the patch to pdn did not solve it, looking [08:42:55] oh, got a kyverno error [08:42:57] Error from server (InternalError): Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/mutate/fail?timeout=10s": context deadline exceeded [08:43:00] looking [08:54:34] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/38 to help debugging the openapi.json intermittent errors now happening on both tools and toolsbeta [08:55:09] dcaro: done [08:55:16] thanks! [09:00:15] hmmm [09:00:16] | calico | chart | calico | calico-0.0.6-20230710081103-dcbbe692 | toolforge-deploy has calico-0.0.7-20240411190528-63954490 | [09:00:19] on toolsbeta [09:00:29] we have a calico < than the one in toolforge-deploy [09:03:33] that's on tools, sorry [09:06:55] you mean tools never got whatever is the our most recent version deployed, only toolsbeta? [09:09:55] yep [09:10:02] toolsbeta has 0.0.7 [09:10:05] | calico | chart | calico | calico-0.0.7-20240411190528-63954490 | | [09:12:31] the change is a pre-commit autoupdate [09:13:00] it'd be cool to have a tool that's just a UI showing the output of toolforge_get_versions for tools and toolsbeta, (and maybe include kubeVersion where applicable) [09:13:25] yep :) [09:14:11] the main issue is that the info of the installed versions in only available for admins (helm list) [09:14:19] so we would have to sort that out somehow [09:19:39] got tekton down alert looking [09:20:40] everything seems a bit shaky today xd [09:23:06] to get us in shape after pto xd [09:44:32] hmpf... the tools prometheus instances are struggling with memory, and getting killed periodically [09:59:01] dns is still an issue [09:59:02] [step-clone] 2024-08-26T09:56:50.943727100Z {"level":"error","ts":1724666210.942934,"caller":"git/git.go:55","msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin --update-head-ok --force ]: exit status 128\nfatal: unable to access 'https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/': Could not resolve host: [09:59:02] gitlab.wikimedia.org\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:55\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\tgithub.com/tektoncd/pipeline/pkg/git/git.go:150\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/git-init/main.go:53\nruntime.main\n\truntime/proc.go:255"} [10:12:16] oh my, all the calico stuff is in a single humongous yaml again... it's going to be very painful to debug [10:12:47] <_joe_> hi, dunno if you've seen https://phabricator.wikimedia.org/T373250 [10:15:21] I had not, looking, thanks for the notice [10:15:43] <_joe_> yeah i see it's been one of those mornings, heh [10:15:44] <_joe_> :( [10:15:56] first day after pto [10:19:39] topranks: can you check the cloudswitch that was misbehaving to see if it's doing weird things again? we are having a bunch or semi-random network issues that might be explained by a wild switch [10:20:05] (it's a stretch as I don't se packages dropped, but well, weirder things have happened) [10:21:06] dcaro: I'm briefly in front of the laptop, in case there is an emergency going on? [10:22:31] random intermittent network issues (DNS inside k8s, timeouts between pods, toolsadmin timing out too, ...) [10:27:18] I don't see anything weird with coredns [10:27:46] coredns logs feel OK in all 3 tools replicas [10:27:49] I didn't either, it's using ~4x the request cpu it sets, but it does not seem to be strucgling [10:27:57] (no restarts or anything) [10:28:22] yeah, and the -k8s-control nodes have CPU headroom, loadavg 2 [10:28:24] I scaled it up to 4 replicas before, and seemed to help shortly (maybe just my imagination, or random luck), and it's back misbehaving [10:28:27] (8 cpu) [10:29:00] striker seems happy now, will monitor for a bit [10:32:58] I'm checking cloudservices nodes [10:33:08] as they are the DNS upstream for some of the work coredns is doing [10:33:32] https://www.irccloud.com/pastebin/nqg7HMnR/ [10:33:44] there seems to be some disk issue in cloudservices1005 [10:34:35] is that increasing or stable? [10:34:40] (it's the same metric as the ceph nodes) [10:35:33] I have only seen that log entry once so far since I started watching the logs [10:36:13] seems stable from previous logs [10:37:01] sda increased by 8 in 3 days [10:37:01] other than that pdns / designate seems to be working as expected [10:37:56] https://www.irccloud.com/pastebin/LLYACkEl/ [10:38:21] I wonder if a rollout restart of coredns would make any difference [10:39:28] I did a rollout restart already [10:39:36] ok, I see 140m ago [10:43:06] I think it times out before [10:43:15] https://www.irccloud.com/pastebin/9vkQPBqw/ [10:43:51] even three retries fails sometimes [10:44:00] https://www.irccloud.com/pastebin/9l69vwzp/ [10:44:12] I wonder if there has been any package updates on the workers recently [10:44:22] or kernel? [10:44:49] maybe, external dns seems to reply reliably [10:45:00] (ex. `I have no name!@shell-1724668943:~$ nslookup github.com 8.8.8.8`) [10:46:15] this also, so k8s specific for sure [10:46:19] `I have no name!@shell-1724668943:~$ nslookup tools-harbor.wmcloud.org 172.20.255.1` [10:46:47] dcaro: just out of meeting [10:46:49] * topranks looking [10:47:46] not the exact same problem anyway - but unlikely it would be I guess [10:48:05] yep, this seems k8s only (though there were a few other issues around, like striker and such) [10:49:09] ok, if there is any specific network thing to check let me know [10:49:40] that 10.96.0.10 IP looks internal to the cluster, it's not in any of the switch routing tables [10:50:20] yep, it's internal yes [10:51:04] I need to go offline now, ping me if the problem escalates into a full outage [10:51:28] sure, thanks, cya [10:55:29] coredns queries per second look ok [10:55:36] https://usercontent.irccloud-cdn.com/file/797u7AGx/image.png [10:56:08] the peaks are more or less the same for the last 30 days, when I scaled up, it went down, but there's no sudden peak in the last days either [11:18:26] something I see is that we do many requests to resolve any non-resolvable name, as we have many domains configured in resolv.conf, not that that would help, but would decrease the load [11:18:44] https://www.irccloud.com/pastebin/KdYwo9ih/ [12:30:06] andrewbogott: also, can you elaborate a bit on what did you do yesterday for the pdns replication? [12:30:14] I'm still struggling to get dns working on k8s [12:35:03] There were two things I did right at the same time. The most obvious is this config setting rename: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1065037 [12:35:47] The other is that I manually reset the domain serial in the database so that axfr would think it was way out of date and sync. I only did that for one domain though so I doubt that fixed anything affecting you. [12:36:14] My tests over the weekend suggested that pdns is fully in sync and that the failure is internal to k8s, are you seeing things that are out of sync or misbehaving within pdns? [12:40:57] ack, thanks, yep, it does not seem related, as the current issues affect all dns entries [12:41:07] they seem to affect only certain workrers though [12:41:26] trying to resolve from k8s-worker-106 works fast and reliably from the coredns pod and from envvars-api pod [12:43:11] seems like we should try rebooting one bad worker and one good one and see if they trade places :) [12:43:29] yep, on it :) [12:43:39] rebooting 104 [12:50:41] I need to go out until our checkin but I can look at the prometheus thing when I'm back. Sorry to vanish! [12:50:42] the reboot did not help :( [12:50:46] https://www.irccloud.com/pastebin/JMRISqs4/ [13:51:44] * dcaro paged harbor down [13:53:52] hmpf... I see the entry in cloud-feed but not on karma ui [13:55:29] grafana does not show any issues either, not sure why it triggered, looking [13:56:34] `Got no data for any component, all might be down or unresponsive, toolforge might be unable to pull images.` might be related to the restarts of prometheusn [13:57:30] I'm going to take the 7 worker nodes that are having dns issues out of the pool, I might create new workers after if I don't find a solution for those [14:01:59] * andrewbogott is back [14:02:14] dcaro: that seems reasonable although it's frustrating that they're misbehaving arbitrarily. [14:02:43] And it does seem like it corresponds to the openstack upgrade? But I can't think why unless they got upset about the service flapping and somehow can never get over it. [14:06:11] I rebooted all of them, and hard stop/started one without any improvement :/ [14:06:33] they are all scattered around cloudvirts too, on different racks and all [14:06:59] so seems bound to the k8s layer to me, maybe just bad timing? [14:09:09] sounds like but it's suspicious! [14:19:02] stashbot was down, we should add an alert for that [14:19:02] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [14:22:54] thanks stashbot xd [14:23:48] clearly not down now [14:24:00] Lucas already restarted it [14:27:32] it's also tricky as it's not just checking that there's pods running, as it gets stuck but is still running, we would have to check if it replies on irc or similar [14:27:49] (maybe the logs?) so we might not have the stats to set the alert right away [14:29:03] * andrewbogott imagines once-per minute conversations on cloud-feed "you there?" "yep, still here" [14:29:35] that'd be doable actually xd [14:54:02] https://www.irccloud.com/pastebin/IT26jBrQ/%20 [15:35:54] > stashbot was down, we should add an alert for that -- y'all want to take over all of my toys ;) [15:37:28] that's because they are useful! [15:37:30] :) [15:39:39] I see some ldap connection drops in stashbot's err log and a handful of dns lookup failures that almost certainly were related to the general k8s dns problems. Nothing exciting though. [16:46:20] andrewbogott: I'm getting `Keystone service is temporarily unavailable. ` on the ui :/ [16:46:35] (horizon) when trying to get the instances for tools [16:46:51] try again? I just restarted some services [16:47:25] ack [16:47:51] seems to work yep [16:50:14] I think I'm going to call it a day for now, I'll leave the current worker being added, and I'll add a couple more workers bit by bit and retake tomorrow [16:50:51] * dcaro off [16:51:03] sgtm. Thanks dcaro