[07:26:24] <godog>	 greetings
[09:17:06] <dcaro>	 Morning
[09:22:23] <dhinus>	 morning!
[10:02:59] <godog>	 I'm seeking feedback/+1s on T417993, thank you
[10:02:59] <stashbot>	 T417993: Request creation of kafka-infrastructure VPS project - https://phabricator.wikimedia.org/T417993
[10:03:59] <volans>	 you already got it :D
[10:04:59] <godog>	 lol great timing, thank you taavi
[10:11:15] <godog>	 mmhhh project creation fails with 504s from opentofu when talking to codfw1dev
[10:11:23] <godog>	 e.g. Error: Error retrieving openstack_dns_zone_v2 7796b60e-7bc6-4cb4-9e4d-dae039d3f912: Expected HTTP response code [200] when accessing [GET https://openstack.codfw1dev.wikimediacloud.org:29001/v2/zones/7796b60e-7bc6-4cb4-9e4d-dae039d3f912], but got 504 instead: {"code": 504, "type": "timeout", "request_id": "req-c6b78bb4-7dfc-46ef-b4f1-1d223a68c2a3"}
[10:11:55] <taavi>	 didn't we change the cookbook to not talk to codfw1dev?
[10:12:31] <taavi>	 ah, only dor the project deletion cookbook :D https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1236783
[10:12:32] <godog>	 not sure offhand, but yes I was a little surprised to see codfw1dev show up when creating eqiad1 projects
[10:12:41] <godog>	 /o\
[10:12:53] <godog>	 I'll do the same for project creation
[10:15:25] <dhinus>	 this was the previous discussion where I advocated against doing that, but given this is _again_ an issue just a month later, +1 for skipping codfw https://phabricator.wikimedia.org/T410265#11498975
[10:15:44] <taavi>	 you may want to also copy https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1236782 to the create project cookbook
[10:16:36] <godog>	 taavi: thank you, will do
[10:16:40] <godog>	 dhinus: ack, I agree
[10:18:46] <dhinus>	 I was wondering why we don't have a firing alert for tofu in codfw being broken, and the answer is https://phabricator.wikimedia.org/T411090#11420249 :P
[10:20:30] <godog>	 sth like https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1242287
[10:20:34] <godog>	 dhinus: heh
[10:23:23] <dhinus>	 in other news, I'm trying to fix tofuinfratest, I cleared a few stuck instances and now I'm left with a puppet-enc error
[10:23:55] <dhinus>	 "Error: Unable to create prefix", https://puppet-enc.cloudinfra.wmcloud.org/v1/tofuinfratest/prefix is returning 500
[10:28:14] <taavi>	 dhinus: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1242291
[10:30:36] <dhinus>	 taavi: awesome thanks, +1d!
[10:33:33] <godog>	 ok I got this MR from the cookbook, I'll go ahead and merge https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/297
[10:34:22] <taavi>	 dhinus: merged and deployed, try now?
[10:34:33] * dhinus retries
[10:35:03] <dhinus>	 cloudvps_puppet_prefix.tofu: Creation complete after 1s [name=tofu-]
[10:47:43] <dhinus>	 tofuinfratest alerts cleared!
[11:03:42] <godog>	 \o/
[11:36:24] <dcaro>	 🎉
[11:36:31] * dcaro lunch
[15:37:26] <dhinus>	 I tried rebuiling lima-kilo and it failed with "toolsbeta-harbor.wmcloud.org/toolforge/openldap:latest: not found"
[15:37:50] <dhinus>	 where is that image coming from?
[15:43:24] <dhinus>	 looks like it was just a copy from upstream: https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/commit/56794d9e9e182e9d646dbb41c61dee1960c24855
[15:43:35] <dhinus>	 not sure why it's not there anymore in toolsbeta-harbor
[15:46:56] <dcaro>	 hmmm, maybe I removed it when I ran the cleanup process manually a few days ago, I think it should not have though
[15:47:07] <dcaro>	 I can push it again
[15:50:52] <dhinus>	 maybe we should add it to docker-registry instead?
[15:51:42] <dhinus>	 I also found releng had the same issue and they created a custom dockerfile https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/170/diffs#be3c8ca70130f5249aff720939df2643e89c0e14
[15:55:02] <dcaro>	 I would prefer not adding more stuff to the docker-registry, and instead consolidate in harbor
[15:55:16] <dcaro>	 (eventually move everything out of there and have one less service/place to maintain)
[15:55:37] <dcaro>	 though I agree that maybe toolsbeta is not the right place for it
[16:10:29] <dhinus>	 dcaro: I see your point, and I'm also not sure where is the best place... let's push it back there for now, to fix the lima-kilo build
[16:13:56] <dcaro>	 hmpf... foxtrot-ldap does not pull the image as is, uses it as base, so I have to find a way to extract it
[16:14:29] <dhinus>	 can you not download it from the bitnami-legacy repo?
[16:23:52] <dcaro>	 oh true, did not test it, give me a sec
[16:25:36] <dcaro>	 dhinus: okok, should be there now, try to create lima-kilo
[16:25:39] <dcaro>	 🤞
[16:25:46] <dhinus>	 thanks!
[16:27:13] <moritzm>	 I've update Bird on the codfw1dev instances of cloudlb and cloudservices to 2.18, it looks all fine to me, unless there's any objections I'd upgrade the prod instances tomorrow?
[16:28:08] <moritzm>	 all the other uses of Bird at Wikimedia are on 2.18 and these are on Trixie/2.17 already, so I don't anticipate any issues
[16:35:42] <dhinus>	 dcaro: the lima-kilo build is working again :)
[16:35:49] <dcaro>	 \o/
[16:36:45] <dcaro>	 pushed it also to tools-harbor, just for backup
[16:38:01] <dcaro>	 eventually we might prefer having foxtrot-ldap already built instead (probably having it's own gitlab repo, with ci/etc.), and use a different openldap (ex. the one from prod) as base
[16:38:26] <dcaro>	 started going that way when openldap got deprecated, but it's trickier than I expected and got distracted
[18:09:52] * dcaro off
[18:09:56] <dcaro>	 cya tomorrow!