[06:52:28] <godog>	 greetings
[10:35:15] <taavi>	 just as a reminder, I will be upgrading kyverno in tools in about half an hour
[11:01:01] <taavi>	 I am starting the Kyverno upgrade
[11:06:02] <taavi>	 upgrade is done, the cookbook is running the test suite now
[11:24:26] <taavi>	 that all was a bit boring even
[11:39:47] <volans>	 boring is nice!
[11:49:45] <taavi>	 if all software would just work as expected, then less people would hire SREs to reliably run their software. so the ideal scenario is software that runs reliably except during business hours
[12:12:00] <godog>	 lol
[12:39:25] <godog>	 taavi: what do you think of https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/63 ?
[12:39:48] <godog>	 can't wait to nuke toolschecker ngl
[12:41:46] <taavi>	 godog: do we know for sure that metric will disappear completely (and not, say, be turned to 0) if the exporter fails to scrape?
[12:42:03] <taavi>	 since it uses count() not sum()
[12:43:59] <taavi>	 but seems fine as a port, I imagine there might be some ways to more reliably check that they're actually in the same cluster with Prometheus but let's not block getting rid of toolschecker on that
[12:44:18] <godog>	 taavi: yes metrics are scraped from etcd itself, no exporter in between thus they won't be there when etcd is down
[12:44:25] <taavi>	 oh right
[12:44:38] <taavi>	 ship it
[12:45:13] <godog>	 \o/ 
[12:45:16] <godog>	 thank you
[12:45:55] <godog>	 interview shortly then I'll proceed with toolschecker removal patches
[13:45:49] <andrewbogott>	 volans: as you predicted, SSH_AUTH_SOCK=/run/keyholder/proxy.sock was the bit missing from my ssh tests yesterday.
[13:45:59] <andrewbogott>	 So the next step is security groups, which requires something like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1298325
[13:49:13] <volans>	 ack
[14:58:50] <bd808>	 toolschecker was such magic when Chase and Madu first built it. Time change. :)
[16:11:42] <godog>	 heheh indeed, that was a pre-prometheus time
[16:12:39] <taavi>	 i'm a bit surprised it wasn't actually just a vm speaking NRPE instead of the http server it has :P
[16:15:44] <godog>	 lolz
[16:25:34] <andrewbogott>	 dduvall, bd808, I just emailed vague instructions about how to test the latest magnum driver. I'm interested in your results if you have the patience to try it... it is definitely the future of magnum although I don't plan to rip out the old driver for a year or so.
[17:16:57] <dduvall>	 andrewbogott: right on. thank you! so assuming it works and we get an LB in front of the master nodes and provision a proxy, how do we ensure the certificate SANs includes the proxy hostname?
[17:18:58] <andrewbogott>	 I confess that this isn't something I've thought about much. Can you tell me a bit more about how traffic is working in this case? I tend to think in simplistic terms: "the proxy does ssl termination and after that it's just http and there aren't any certs needed" but sounds like that doesn't apply in this case.
[17:19:44] <bd808>	 dduvall: I think that you be fronting that LB with the existing wmcloud.org proxy. It would terminate TLS for the client, but then could be configured to re-encrypt traffic to the LB which would work just fine with a cert missing the public name's SAN.
[17:20:20] <bd808>	 I had that working at one point, but then took the wmcloud proxy out because it is anohter moving thing to break
[17:20:31] <andrewbogott>	 It's also the case that we can definitely just give you a floating IP and let you manage things yourself if that somehow makes life easy.
[17:21:21] <andrewbogott>	 We are trying/hoping to have a small amount of scraper protection in front of the proxy. We don't have that /now/ but that would be one advantage to keeping it in the mix in the long run.
[17:24:39] <taavi>	 if magnum needs a public v4 to be used properly then that's a bug in magnum that needs to be fixed IHMO
[17:25:34] <dduvall>	 a floating IP would for now make it easier (and i've been banging my head on zuul for a while now and am close to burnout tbh)
[17:25:50] <andrewbogott>	 taavi: it doesn't for the use cases that I'm expecting... but I don't think I understand what dan needs/is doing.
[17:25:52] <dduvall>	 i wish we could just add a label or something to get another hostname into the SANs but that doesn't seem possible
[17:26:18] <andrewbogott>	 you can certainly add a second proxy with a second name but the same backend...
[17:26:37] <dduvall>	 i mean i can try the new cluster-api magnum driver but i'm knee deep in trying new and somewhat broken things
[17:27:37] <dduvall>	 i quite regret upgrading us to zuul 14.x w/ the new zuul launcher as is. it seems like they're still ironing out problems (though it seems like nodepool had this same problem)
[17:31:44] <dduvall>	 bd808: i suppose i can test the `client <-(tls)-> proxy <-(tls)-> master node` setup now even without the octavia lb, yeah?
[17:34:00] <taavi>	 bd808: the wmcloud.org proxy breaks TLS client authentication if you're using that with k8s
[18:01:49] <dduvall>	 chatgpt told me that the cluster-api driver should allow us to add an additional SANs entry, but i don't really believe that yet :D
[18:16:35] <andrewbogott>	 yeah, I'm digging in the docs to see how one would do that...
[18:24:31] <andrewbogott>	 ...and now digging in the code
[18:41:00] <andrewbogott>	 dduvall: are you already using labels in your magnum template?
[18:41:29] <andrewbogott>	 If yes, please try adding a 'api_server_cert_sans=comma,delimited,list,of,names' label and see if that does something useful?
[18:42:03] <dduvall>	 andrewbogott: yeah, we have a number of labels
[18:42:13] <dduvall>	 should that work with heat too or just cluster-api?
[18:42:16] <andrewbogott>	 I predict that that will work with the new driver but not with the heat driver. But it /might/ also have worked with heat, not sure
[18:42:21] <dduvall>	 ah got it
[18:42:29] <andrewbogott>	 I'm reading the cluster-api code now, I see it implemented there
[18:42:36] <andrewbogott>	 but I don't know if it's a carry-over from heat or a new feature
[18:43:15] * andrewbogott tries to find a checkout of the heat code...
[18:43:48] <dduvall>	 i grepped the stable/2026.1 branch of magnum but no dice. i can try out the new driver later today
[18:44:07] <andrewbogott>	 yes, I don't see that label implemented in the heat driver
[18:44:26] <andrewbogott>	 So I guess this motivates you to try cluster-api which suites me :) I hope it's not too painful.
[18:45:02] <andrewbogott>	 are you using nginx-ingress?
[18:45:26] <dduvall>	 within the cluster?
[18:45:29] <andrewbogott>	 yeah
[18:45:30] <dduvall>	 nothing yet
[18:46:11] <andrewbogott>	 ok, good, it doesn't exist in k8s 1.34.8 -- an issue for PAWS but not for you
[18:46:31] <dduvall>	 i don't think we'll need ingress. zuul will create its own namespaces/pods for each job. we'll likely need buildkitd in there to drive former pipelinelib type workloads
[18:46:38] <dduvall>	 ah ok. good to know!
[18:48:10] <dduvall>	 re: not in 1.34 oh yeah, we already dealt with that in the gitlab-cloud-runner cluster. we're using traefik now
[18:49:01] <andrewbogott>	 cool, that may be in my future
[18:49:10] <andrewbogott>	 Or the future of <future owner of paws>
[21:02:34] <dduvall>	 andrewbogott: is there an equivalent of https://docs.openstack.org/magnum/latest/user/#supported-versions somewhere for ubuntu/cluster-api?
[21:03:12] <andrewbogott>	 not as far as I could find, I cajoled one of the devs to paste their personal chart into irc let me see if I can still find it...
[21:03:13] <dduvall>	 i mean i can experiment but iirc it took some trial and error to find the right incantation of labels for the current setup
[21:03:23] <dduvall>	 oh that would be awesome
[21:03:27] <dduvall>	 (a working example)
[21:04:06] <andrewbogott>	 The template pasted in my email /should/ be a working example, it built a paws cluster in codfw1dev
[21:04:27] <dduvall>	 oh geez yeah sorry. i hadn't scroll that far yet :)
[21:04:32] <dduvall>	 ty!
[21:08:34] <andrewbogott>	 huh, I literally didn't think it was possible to format something this badly in wikitexth https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Magnum_setup#Compatibility_matrix
[21:09:09] <andrewbogott>	 any suggestions how to get this onto that page legibly? https://www.irccloud.com/pastebin/n07AKJwG/mcapi%20support%20matrix
[21:10:43] <bd808>	 andrewbogott: make a real table instead of an ascii spew
[21:10:55] <andrewbogott>	 that sounds like work, but ok :(
[21:11:04] * bd808 will fix it
[21:11:28] <andrewbogott>	 nooo I wasn't trying to nerd-snipe, I'll do it
[21:12:07] * bd808 sees H1s and starts eye twitching
[21:12:36] <andrewbogott>	 you can stand down, I'm already filling in the rows
[21:19:19] <andrewbogott>	 dduvall: I'm about to go collect my CSA radishes. Are you mostly unblocked for the moment?
[21:20:35] <andrewbogott>	 If you want a tofu version of that template, this is it give or take a setting or two:  https://github.com/toolforge/paws/blob/main/tofu/135a.tf-magnum-capi
[21:21:52] <dduvall>	 andrewbogott: oh yum! i just got an error but i can poke at it for now
[21:22:17] <dduvall>	 `| status_reason        | failed to extract field Cluster.labels`
[21:22:28] <dduvall>	 checking my labels now
[21:23:09] <andrewbogott>	 I'll be back in a bit, and around off and on this evening.
[21:23:20] <dduvall>	 great, thanks!
[21:31:20] <dduvall>	 yikes, getting `DELETE_FAILED` now with the same `failed to extract field Cluster.labels` status
[21:37:54] <andrewbogott>	 that's not good
[21:38:38] <andrewbogott>	 this is zuul-k8s-v134 ?
[21:42:32] <andrewbogott>	 the logs say "TypeError: 'str' object is not an instance of 'bool'" near that error, seems telling
[21:45:44] <andrewbogott>	 TypeError: failed to extract field ClusterLabels.cinder_csi_enabled
[21:50:12] <andrewbogott>	 since the labels are driver-specific I don't think the APIs know enough about them to validate. I think you should either remove that label entirely (I think it defaults to true) or try True instead of true. 
[21:50:25] <andrewbogott>	 Sorry, this is clearly going to be a real pain to get right.
[21:50:36] * andrewbogott really going for radishes now
[22:32:11] <dduvall>	 oh maybe it's sensitive to the types in some of the flag labels like `master_lb_enabled` whereas the heat stack templates aren't? i see your tofu uses string values for those
[22:32:31] <dduvall>	 and ours uses bool
[22:41:33] <andrewbogott>	 oh yeah, that could definitely be it.
[22:41:42] <andrewbogott>	 Are you currently wedged because it refuses to delete?
[22:46:07] <dduvall>	 yeah :(
[22:46:42] <dduvall>	 can you throw your radishes at it please? mario 2 style?
[22:47:30] <dduvall>	 sorry, super mario 2 (Nintendo USA)
[22:55:14] <dduvall>	 i need to pick up my girls from camp, but yeah andrewbogott if you get around to deleting that cluster for me i can retry with string values for those labels
[22:55:23] <dduvall>	 thanks for your help!
[22:55:44] <andrewbogott>	 yep, I'll see what I can figure out