[02:40:16] I've restored egress networks to all projects except for mobileappsperformance which is some kind of tofu-related edge case that I haven't investigated yet. [02:40:38] well, not to the bullseye projects [09:00:46] morning! [09:54:02] taavi: https://phabricator.wikimedia.org/T442510 TIL we can't login in emergency mode [09:54:14] lovely [10:20:00] FYI I'll resolve two acked wmcs pages [10:20:17] https://portal.victorops.com/ui/wikimedia/incident/8634/details https://portal.victorops.com/ui/wikimedia/incident/8633/details from a week ago [12:41:48] andrewbogott: it seems like `maps-wma3` (in the `maps`) project is missing from DNS entirely? can you tell if it ever worked or is that instance just entirely borked? [13:26:07] taavi: I don't know what's up with maps-wma3. Puppet seems to work on it so it's not 100% broken... [13:26:20] Somewhere there should be a task about maps migrating off bullseye that might explain [13:26:40] I believe wma3 is supposed to replace the bullseye wma2 instance :/ [13:26:52] right [13:27:29] I can only find T432955 which is only about a different maps VM [13:27:29] T432955: Migrate maps-osmdb from bullseye to .. - https://phabricator.wikimedia.org/T432955 [13:28:28] andrewbogott: for paws, eqiad should be able to handle capi clusters right? (I'm getting "Cluster type (vm, Unset, kubernetes) not supported" from openstack when running tofu on the 135a.tf file) [13:29:10] yes but if you're using the same template as in codfw1dev then you need to update the worker base image [13:30:00] best is to use ubuntu-22.04-v1.36.4-for-magnum [13:30:11] and then adjust the k8s tag and version to 1.36 [13:30:14] okok, I'll rename the cluster to 136 then [13:30:41] yeah... I thought I had standardized that across eqiad and codfw1dev but probably after I wrote that template [13:31:03] did you get the nginx/istio bits sorted? [13:31:29] I have to test it, as in codfw was not able to do it properly [13:31:31] I looked at it but found nginx in more places than I was expecting and backed away :/ [13:31:37] sure [13:32:13] I would leave the paws template mostly as it is, but you can see a working template example with "openstack coe cluster template show f486aece-d895-4822-b6cb-ee78e4e7da42" [13:34:00] dcaro: I'm not totally sure how to actually test a new paws deployment without making it live for everyone. There are some hardcoded URLs in there so when I try to point a different domain to a test cluster it always manages to redirect to the prod one. [13:34:09] On the other hand, today is probably a great day to fix that :) [13:34:25] (I mean, when I ran into the issue originally, that would've been a better day to fix it) [13:34:33] andrewbogott: paws is fully down at the moment, bringing an experimental but up to date cluster up is totally fine ihmo [13:35:01] True, I guess it might be broken but probably not in a way that will destroy anything of value [13:36:22] btw, thanks for restoring Horizon + api access. It was getting late and I didn't want to open the floodgates and immediately leave things unattended. [13:39:00] dcaro: the magnum-capi people talk a lot about how fast their implementation is... I guess everything's relative :/ [13:41:23] taavi: I noticed your comment about imagor a while ago -- did you already compare that wiki page against "openstack project list" or shall I do that while we wait for magnum? [13:42:24] please do [14:01:47] wiki page and 'openstack project list' are now reconciled. I didn't add any 'todo' projects but did beef up the 'fully restored' list quite a bit. [14:07:28] dcaro: were my comments about the paws magnum template above baffling? Want me to work on getting the magnum bits working in eqiad1? [14:07:51] I think I'm ok, let me gather up a patch for you to review [14:08:22] ok [14:08:22] there's a 1.36 cluster for quarry, but quarry code does not register it, where can I see that code? [14:08:32] (/me is looking for the exact tags) [14:08:42] like `cloud_provider_tag` [14:09:28] https://github.com/toolforge/quarry/pull/104 and https://github.com/toolforge/quarry/pull/105 [14:09:37] nice 👍 [14:09:41] which I will merge shortly since they're already deployed [14:10:12] yep, that's useful [14:10:23] network and such should be the same also right? [14:11:19] I think so [14:12:05] Those templates are inherited from Vivian and there are a lot of things in there that I think are overspecified... I weeded out some bits but not others so you may see some disagreement about tags between the two projects [14:12:25] okok [14:12:26] s/tags/labels/ [14:13:28] we're also in some awkward half-migrated state between github and gitlab, going to make a task about that right now [14:14:00] on quarry you mean? [14:16:21] Um... I think paws and quarry both have partial migrations to gitlab that were never used. So you can ignore me on this topic, just thinking out loud. Github is canonical for paws and quarry code, today. [14:16:29] andrewbogott: https://github.com/toolforge/paws/pull/537/changes#diff-563462594e90389f80719af1306606b15cd65ffd58814872fec0fb5c5d12b845 the 136 cluster def [14:16:48] ack, yep, we half-started but never really xd [14:18:05] that looks right to me [14:18:17] ack, /me trying to deploy [14:18:32] andrewbogott: fyi. on codfw1dev I was unable to ssh to the worker :/, the key was getting rejected [14:19:03] mismatched kube_tag is my #1 cause of failed magnum deploys [14:19:21] OK, we'll see if ansible has better luck in eqiad1 :/ [14:20:29] oops, messed it up `abogottcodfw1dev` key [14:20:31] looking [14:20:50] it should be paws-magnum-vm right? [14:21:05] I think so? Same as it was in the older template [14:21:33] Sorry you're having to deal with my test-dev-just-for-andrew template as a baseline [14:22:41] no problem [14:22:57] will it update the template? or should I remove it before retry? [14:23:33] it seems it will try [14:23:59] yeah, tofu manages the template [14:25:37] okok, promising :) [14:26:14] that that runbook about tailing the capi/capo worker pods turn out to make sense and/or be useful? [14:29:08] I did not need to use it yet :), codfw came up and was just checking the k8s there, let me try [14:31:12] oops: 'Failed to create security group k8s-cluster-magnum-system-kube-53vmb-secgroup-controlplane' [14:31:33] because it's not allowed :/ [14:31:35] hmmm [14:32:02] btw. yep, the runbook has been useful :) [14:46:25] I also see the error `"External network found"` in the logs, looking [14:47:46] weird, the network is there `wan-transport-eqiad` (same as quarry) [14:47:56] anyhow, we can retry once the secgroup is done [14:49:20] is that the same network the old template was using? [14:49:44] for external yes [14:49:51] (same that quarry uses in the patch) [14:49:54] hm [14:50:05] policy patch is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359288 -- first draft at least [14:51:42] +1d [14:52:40] why does pcc say noop, shouldn't it show the file change? [14:54:02] 👀 [14:55:39] hmm... the file is there [14:55:47] https://www.irccloud.com/pastebin/0rxatUvl/ [14:57:55] maybe for files it does not show the changes? [14:57:56] yeah, maybe diffs within static files don't appear? [14:58:13] the change definitely deployed [14:58:17] try paws deploy again? [14:58:39] 👍 [14:59:09] did you re-create the paws application credential to have the new role already? [15:00:17] nope xd `{\"NeutronError\": {\"type\": \"PolicyNotAuthorized\"` [15:00:52] how do I recreate it? [15:01:43] oh, hang on, I was thinking the magnum service users was doing this... [15:01:49] haven't added the role to paws user yet [15:02:35] well... hm [15:02:46] taavi, what user are those app credentials from? [15:03:07] hopefully a dedicated service account but no idea in practice [15:06:15] that might need changing inside magnum capi itself no? [15:06:34] I'm still digging. I don't think so though [15:06:55] trying to find the app credential ID, lmk if you find it first [15:07:39] found I think... [15:08:52] ok, service user is paws-infra [15:09:06] I will add the role to paws-infra and we'll see if the new role is automatically delegated. If not will make new creds [15:09:35] dcaro: try again? [15:11:00] it is not, application credentials are scoped to specific roles specified when creating the credential [15:11:46] even if they're... Oh right, I guess the scary flag is about re-delegation, not about live updating. [15:11:58] ok, stay tuned... [15:14:18] OMG pwstore worked on the first try! That has never happened happened to me before, feels like I should buy a lottery ticket [15:20:15] dcaro: I've dropped the new creds into /root/paws/tofu/ [15:20:18] want to try another deploy? [15:20:52] yep, on it [15:22:29] \o/ `Created security group k8s-cluster-magnum-system-kube-5hk3b-secgroup-controlplane` [15:22:49] progress [15:23:36] https://github.com/toolforge/paws/pull/539 [15:24:00] nodes are coming up! [15:24:09] Boy it turns out that when you break a fundamental part of cloud operations that a lot of things get broken :( [15:24:41] Usually there's a long delay after the controller node comes up... once a worker appears then you know magnum is happy [15:27:53] how likely are we to want to redeploy quarry while secgroups are still locked down? [15:28:33] I'd say quite likely? (as it will be a while until everyone has time to go through their cloudvps no?) [15:29:03] please don't worry about quarry just quite yet :D [15:29:31] ok. We'll need the same role+cred update there before we can deploy [15:33:47] it seems to be stuck, probably with the loaad balancer? [15:35:17] it tends to take quite a while... maybe just be slow [15:35:28] unless it's reporting an error state [15:35:59] it says timeout waiting for load balancer, but it seems to keep retrying [15:36:14] lb was updated at '2026-10-27T15:35:37' [15:36:16] which is pretty recent [15:36:21] but timeout doesn't sound great... [15:36:31] oh, volume quota reached [15:37:20] ok, that's probably in tofu, I'll make a patch [15:37:47] should I delete the servers for 127d? [15:38:11] we have two clusters that we probably will not bring up [15:38:29] are there two 127 clusters up? [15:38:43] yep, well powered off [15:38:59] Sure, delete whichever one of them is the right one to delete [15:39:04] hahahaha [15:39:16] fixing quotas the right way is difficult because gitlab is not letting me fetch [15:40:13] ack, this will free space right? [15:41:24] it should! there's a big volume attached to every worker [15:44:08] cluster up, running ansible... [15:45:25] oh, it recovered as soon as the quota issue was fixed? [15:45:39] good job to capi for retrying forever [15:46:26] ...can other people also not fetch from gitlab just now? [15:48:08] yep, it kind of just continued trying [15:48:36] And can't fetch either from gitlab (hanging) [15:49:00] ok, I won't try to fix my laptop then, thanks for checking [15:49:38] hmmm... tofu only created one worker though :/ [15:49:50] I mean, magnum [15:50:21] wait no, there's 4 , but not showing on k8s, maybe they are still starting up? [15:51:07] https://www.irccloud.com/pastebin/xpYM8p8W/ [15:51:10] hmm :/ [15:51:58] {"kube-5hk3b-default-worker-h42m4-djnsz-t6w4w.Ready":"False","kube-5hk3b-xz8s8-5p7h7.Ready":"True","kube-5hk3b-default-worker-h42m4-djnsz-p594c.Ready":"False","kube-5hk3b-default-worker-h42m4-djnsz-975kw.Ready":"False","kube-5hk3b-default-worker-h42m4-djnsz-st5b2.Ready":"False","kube-5hk3b-default-worker-h42m4-djnsz-mw7dd.Ready":"True","api":"ok"} [15:52:04] yep, it kinda failed [15:52:51] tofu did not care though [15:52:53] openstack_containerinfra_cluster_v1.k8s_136a: Creation complete after 15m17s [id=da900116-6d28-4f54-932c-4c9d9ca8d999] [15:53:01] can I force it to recreate the cluster? [15:53:57] let me try to resize to 1, then to 5 again [15:54:14] I wonder if that's a delayed effect from the quota failure? [15:54:32] Seems like the additional drive didn't get created or attached properly [15:54:47] If there's plenty of quota space now I guess I would just redeploy [15:54:53] long wait though :/ [15:55:10] well, you're about to be in a meeting so the wait won't make much difference [15:59:15] hello! do those two cludcontrol-dev hosts have the proper netbox status "planned", or should they be active? https://netbox.wikimedia.org/extras/scripts/results/411839/ [16:00:14] andrewbogott: to delete the cluster I just move the tf file and let tofu handle it right? [16:00:36] yes [16:00:56] unless you want to force a redeploy of the exact same thing, then I would do 'coe cluster delete' [16:34:06] andrewbogott: it failed this time to create any workers :/ [16:34:21] it seems it's trying to delete volumes and failing ata it [16:34:23] *at it [16:34:40] ok. Give me a minute and I'll try to clean things up... [16:35:13] oh no, the workers aare up now [16:35:14] :/ [16:35:18] * andrewbogott stands back [16:36:01] looking, still some issue with the volumes `[ 50.902900] cloud-init[539]: Stderr: sfdisk: /dev/vdb: does not contain a recognized partition table` [16:37:05] have you seen that before? [16:37:10] I haven't [16:37:26] although naturally I suspect repeatability with anything to do with volume labels [16:37:37] oh, rebooted the worker and now it did not complain [16:37:42] maybe wait a bit, it might be doing something out of order [16:38:24] oh, wait, this is an ansible error, not a magnum thing? [16:38:40] nono, magnum [16:38:54] it's hard to track as the nodes don't have 'paws' on their names anaymore :/ [16:40:17] I see a node still in build state, not sure what that means [16:41:22] oh, it just deleted it and recreated it [16:42:32] capi is all about reconciliation loops [16:42:55] xd, eventually consistent right? [16:43:07] sometimes eventually [16:43:20] for very very big values of eventually [16:43:47] I'll wait then a bit more [16:44:44] If you wind up starting over, there's a new base image to use [16:44:46] | 0e56f17c-bb82-4b63-a4b6-ed25455871ce | ubuntu-22.04-v1.36.5-for-magnum | active | [16:44:53] nice :), thanks! [16:44:54] with fresh package installs [16:45:00] note that you'll need to change the k8s tag if you use that [16:45:07] btw. can I use the name of the image in tofu? or should I use the id? [16:45:17] pretty sure the name works too [16:45:21] in the template def [16:46:25] awesome [16:47:28] if/when you need to hand this off to me entirely... will it be obvious if the ingress bits are working properly? Is my test just "a notebook appears and can run a line of code"? Or are there subtleties? [16:50:33] there might be subtelties, so we have to disable the public side [16:50:51] ok [16:51:07] thoughts about how to test w/out making everything public? [16:51:22] it's a service, we can just not add the webproxy [16:51:27] (not adjust it) [16:51:47] public-paws.wmcloud.org [16:51:55] xd [16:52:51] and then test with... curl? [16:53:02] prometheus might not work, but we can deal with that later (iirc it was not working already before) [16:53:11] yep, that would be doable, or ssh tunel [16:53:51] so if it allows to log in, create a notebook, and run a silly python script, I think we are in really good shape [16:54:08] * andrewbogott does not know how to create a notebook via curl [16:54:20] xd, yep, might be tricky [16:54:22] but there's also the security issue right? So probably we aren't releasing this today [16:54:30] yep, we might not [16:54:40] the security issue is the public-paws thing [16:54:50] (so not hard to just block for now) [16:55:08] magnum says "CREATE_COMPLETE | UNHEALTHY" [16:55:16] which imo 'unhealthy' just means 'there is only one control node' [16:55:19] which is the case here right? [16:55:40] um... in my experience, not my opinion [16:55:50] it seems to be in a loop creating/deleting that vm [16:55:56] https://www.irccloud.com/pastebin/jiYUAzqZ/ [16:56:13] ugh [16:56:22] the control node and 4 workers look ok [16:56:40] but ansible must've already fired [16:56:45] so probably you can test and not worry about that one worker [16:56:57] csi-cinder-nodeplugin is failing to start, so I'll look into that [16:57:03] yep [16:57:14] ansible just went through without problems [16:57:24] (stuff is not really running though) [16:57:59] oh, hub pod fails to mount `MountVolume.SetUp failed for volume "homes" : mount failed: exit status 32` [16:58:17] that's nfs isn't it? [16:58:21] that might be a side effect of the csi plugin [16:58:22] yep [17:00:36] def thought I got further than this in codfw1dev but maybe I was fooling myself [17:01:03] well, quarry uses nfs and it works there [17:01:17] should be roughly the same setup -- attached cinder volumes + nfs on every worker [17:03:11] oh, the last worker came online [17:03:39] I think this could be a secgroup thing but you should check my work. [17:03:51] nfs allows access to the default secgroup but magnum workers aren't in that group [17:04:09] I'm going to add 'default' to some workers and let's see if that does something... [17:04:13] healthy now xd [17:04:27] ohhh, yep, that might be the issue [17:05:47] let me know when you do and I'll kickstart the pods [17:05:51] ok, done [17:06:00] oh, yep, the pod started itself xd [17:06:11] if that works then we will need to understand why this works in quarry [17:07:15] did you add all the nodes? [17:07:35] I tried to add 'default' to all the kube- vms [17:07:38] can double check [17:08:36] I htink I double deployed openstack csi plugin, and that's why it's getting stuck [17:08:38] xd [17:22:40] I think I fixed the secgroup thing for new cluster builds, although it might have races [17:25:28] andrewbogott: does magnum install the openstack csi plugin already? [17:25:42] yes, as far as I know it does [17:25:56] xd [17:26:03] okok, then we don't need it anymore in paws helm stuff [17:26:03] were we installing it with ansible after the fact? [17:26:19] I'm not 100% sure but that would explain the double install [17:26:56] yep [17:27:54] the magnum one is already good right? (credentials etc.) [17:30:03] okok, I'll try rebuilding it once more, then it's all yours [17:31:32] (if you have time of course, I'll catch up tomorrow) [17:31:33] using the new image too? [17:31:36] yep [17:31:44] ok [17:34:18] oh, broke something :/ '{"errors": [{"request_id": "", "code": "client", "status": 400, "title": "Cluster type (vm, debian, kubernetes) not supported", "detail": "Cluster type (vm, debian, kubernetes)' [17:35:02] maybe the image? [17:35:20] try the ID instead of the name I guess [17:36:16] oh wait I see it, one second [17:37:04] did not work, I think I fixed that before xd [17:37:30] I had it set to debian instead of ubuntu [17:37:31] try again? [17:38:01] oh, okok [17:38:15] you're testing with all these changes? https://github.com/toolforge/paws/pull/537 [17:38:31] yep, got thorugh :) [17:38:41] yep, the whole stack [17:38:52] so far all the commits are kinda needed to make it work [17:39:30] ok, and deploying from /root/paws on the bastion? [17:39:38] yes [17:39:45] there's an extra file there that was never in the repo [17:39:54] the 127e.tf, that is the current cluster [17:40:04] ok [17:40:19] well maybe everything will work on this attempt and then we can just merge that whole thing [17:40:28] just made a copy of it in /root in case I do git clean [17:40:32] xd [17:40:33] yep [17:40:35] hopefully [17:40:40] it does 3 things mostly [17:40:46] * rebuild all images with latest stable versions [17:40:56] * move from nginx to istio [17:41:05] * move to k8s 1.36 [17:41:26] all kinda need each other (can't upgrade without k8s, can k8s without istio, can't istio without upgrading) [17:42:38] * andrewbogott nods [17:59:39] ansible timed out trying to deploy istio, last time it was because the workers were not around yet [17:59:50] yep xd [17:59:52] https://www.irccloud.com/pastebin/GC29pRTJ/ [18:00:02] * dcaro patience [18:00:52] not sure how much I trust this: CREATE_COMPLETE [18:01:43] it's a good start! [18:02:04] Doesn't ansible wait until tofu says create_complete? [18:06:48] yep, but tofu just says it's ok way before the cluster is up and running [18:06:52] (that happened before too) [18:07:03] maybe a difference on how the magnum backend reports back? [18:07:11] huh. Does tofu return as soon as the creation is dispatched? [18:07:32] kinda, not exactly, but almost [18:07:39] I thought it blocked until CREATE_COMPLETE [18:07:43] https://www.irccloud.com/pastebin/L9JT7NNj/ [18:07:47] it took 14 min [18:08:07] ok. so does that break everything? Or can you redeploy after and just get the ansible parts? [18:08:30] you can just do the ansible bits later yep [18:08:52] (manually, just check the script, you need to source the venv, I should probably just add the option to the script xd) [18:08:53] ok [18:09:08] I bet that if you just re-run the deploy script it'll do a tofu no-op and procede [18:09:10] https://www.irccloud.com/pastebin/L4gCbe9T/ [18:09:11] proceed ? [18:09:16] it just needs patience xd [18:09:36] let me try yep [18:12:43] sorry, I was wondering how to spell the word, not telling you you needed to procede/proceed [18:12:57] (it's proceed, says the internet) [18:14:53] all pods are up and running so far :) [18:15:00] ansible finished [18:15:28] "might" be working xd [18:17:27] hmm.... thinking how to test it now [18:20:22] there might be sec groups messing about? [18:20:39] is nfs still not working? [18:21:19] I think it is yes [18:21:34] but for ingress stuff (first time I deal with magnum lb) [18:22:02] For now just bypass the lb and connect straight to the node [18:22:04] that should work [18:23:02] that does something, looking [18:23:07] In a perfect world we could create a novaproxy that restricted access to an IP range and limit it to just your apartment. [18:23:16] I wonder how hard that would be to mock up... [18:23:29] I guess it couldn't really be done with security groups [18:23:58] but you could attach a floating IP to that k8s node and then open that node to your home IP [18:24:07] assuming your home IP is somewhat static and not behind a giant nat [18:24:20] promissing [18:24:30] I'm using ssh tunnels and such [18:24:53] https://www.irccloud.com/pastebin/bhxqAH4Q/ [18:24:55] ok! that's better than my clumsy idea [18:25:40] no idea if that 404 is good or bad [18:31:12] That was good [18:31:31] Clodflare has blocked me for irccloud, so using the phone [18:31:47] (it does not like me using SSH proxies) [18:31:48] what the heck, cloudflare [18:32:11] I can get the jupyterhub login page, but of course oauth does not work [18:32:16] In the future only criminals will have tunnels [18:32:41] 'of course' because you're doing this with curl instead of a browser? [18:32:59] oh, and because of the dns name I suppose [18:36:08] I'm back! [18:36:42] so jupyter is up and running, it shows the login page, but can't really log-in easily without external access (as the callback will need to be to the page) [18:36:56] unless I do some really hackish things :/ [18:47:07] \o/ [18:47:10] https://usercontent.irccloud-cdn.com/file/14aMx1QW/image.png [18:47:38] Server launch is taking longer than expected. Please be patient! Current time spent waiting: 60 seconds. [18:48:11] it seems to be just slow so far (in k8s too, says pulling) [18:48:43] there's some issue with the cpu metrics (not related) [18:49:13] it came up [18:49:31] https://usercontent.irccloud-cdn.com/file/ESdxG5CH/image.png [18:49:37] fantastic! [18:50:55] a silly python notebook worked [18:50:57] https://usercontent.irccloud-cdn.com/file/9Cju4dKU/image.png [18:52:13] I'm not sure now what was actually availabel xd [18:52:24] openrefine, r, julia, python all are there [18:53:13] openrefine seems not to work though (503) [18:53:59] that's weird, why would it be any different [18:54:05] oh, it showed up now [18:54:09] maybe it just took a sec to start [19:47:30] taavi: earlier you said you wanted me to hold off on quarry, does that mean you have other things in flight and I shouldn't redeploy? [19:47:42] why would you redeploy quarry? [19:48:58] 1) to run it on the same worker image as paws 2) to get pending patches merged and make sure that what we're running is the same as what's in git [19:49:01] not urgent, just cleanup [19:49:32] ah, sure [19:49:47] so nothing blocking that on your end? [19:49:52] my point earlier was just 'quarry is working fine for now, please focus on the service that is broken and that people are asking about' :D [19:49:58] so go ahead [19:50:20] ah, fair enough :) That's what I thought but wanted to make sure there wasn't some new security disaster around the corner :) [21:34:43] The quarry deployment is now aligned with the git main branch, deploys cleanly with modern magnum, and seems to work. Stepping away before my luck runs out.