[06:42:53] morning! looking for a review of https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221 [07:28:52] morning, wrote a comment there, I might be misunderstanding what the patch does xd [07:30:41] uhh excellent question [07:31:17] the security groups in that project are a bit of a mess tbh [07:31:37] ah, the 'allow proxy traffic' rules are in the default group. sigh [07:33:03] yep, security groups has been something we have not been tidying up much xd [07:33:21] yeah, and that's what i'm trying to change :D [07:33:39] +100 :) [07:33:58] gtg to the doctor though, I'll be back in a bit [07:34:11] ok, patch updated to "import" the current rules, except with the new IP blocks [07:34:23] and then I'll send a follow-up to add the traffic-allowing patches to that group [07:34:54] ttyl then :-) [07:38:51] LGTM now, missing the plan, +1 if the plan looks ok too [07:38:54] cya! [07:47:05] next up: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223 [08:15:11] taavi: +1 [08:20:45] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139782 [08:34:48] moved the floating ip to maps-proxy-5. still waiting to get that puppet patch merged before adding AAAA records [08:43:26] +1'd [08:46:29] sigh, didn't work [08:46:29] Apr 29 08:46:01 maps-proxy-6 nginx[41744]: 2025/04/29 08:46:01 [emerg] 41744#41744: duplicate listen options for [::]:443 in /etc/nginx/sites-enabled/proxies:76 [08:47:16] :-S [08:47:20] ugly syntax [08:47:31] maybe chatgpt can help you get it right [08:47:32] anyway, I disabled puppet on the active node so we have some time to fix [08:50:43] i've no plans to touch those environmental catastrophes [08:50:53] :-) [08:51:32] tested that this actually works: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139786/ [08:52:07] +1 [08:59:59] uh, no, now that's binding on v6 only [09:00:02] sigh [09:03:19] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139791 [09:03:31] (i'll merge once CI passes) [09:07:19] finally looks good [09:07:23] so rolling out to the active node [09:19:59] 👍 [09:45:59] fun numbers: for tools-legacy-redirector, about 1/3 of the traffic is IPv6. for maps-proxy, right now that's about 1/5, although the DNS TTLs on there mean it's still probably increasing [09:46:14] one-liner for counting: awk '{print ($1 ~ /:/ ? "IPv6" : "IPv4")}' /var/log/nginx/*.log | sort | uniq -c [10:06:21] sigh T392889 [10:06:22] T392889: replication broken on cloudinfra-db04 - https://phabricator.wikimedia.org/T392889 [10:09:28] :-( [10:09:52] I guess that project may also benefit from a bit of additional tofu [10:25:02] wonder whether it'd be possible to do https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Keepalived#Neutron_configuration via tofu [10:42:12] taavi: [10:42:18] allocating a port is trivial in tofu [10:42:20] https://search.opentofu.org/provider/terraform-provider-openstack/openstack/latest/docs/resources/networking_port_v2 [10:42:26] and so is configuring the allowed-address [10:42:33] so yes, it should be possible [10:58:14] I guess we only lack the puppet side of things, might be doable with the puppet enc only [11:00:41] proxy-5 alert is me, silencing [11:15:48] please review: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:46:31] taavi: +1d [11:47:07] taavi: beware that VMs cannot be created using tofu-infra (they will go to the admin tenant) [11:47:30] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/tofu-infra#Virtual_machine_instances_can_only_be_created_in_the_project_specified_in_the_openstack_auth [11:47:40] ack. I did create these using the cookbook anyway [11:48:40] ok [12:48:15] it might be worth adding some abstraction in tofu-infra so that adding allow 0.0.0.0/0 and ::/0 don't require two copy-pasted rules, and allowing all of cloud vps doesn't take two rules with copy-pasted source ip data [12:48:29] anyway, for now, https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225 [13:00:29] taavi: +1d [13:19:00] somehow both of the new instances think they're the keepalived primary for the v6 vip :( [13:25:33] a-ha, I think it's trying to use its link local address as its own unicast address [13:31:46] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139857 [13:41:35] I think I saw something similar in cloudgw servers [13:44:06] and of course puppet will sometimes pick the VIP as $facts['networking']['ip6'] :// [14:59:09] arturo: what do you think about this approach for selecting the unicast source ipv6? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139877 [14:59:49] taavi: maybe hiera [15:01:02] i mean, ihmo that patch shows a reliable way to detect it automatically, and having it in hiera will just be one more thing to do manually when provisioning hosts [15:25:03] taavi: oh, I see now. Yeah, looks good [15:26:38] +1d [15:26:52] also, you may want to decouple the logic into a function or something, and add testcases [15:33:25] quick review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1139887 , I'm configuring a yubikey for my ssh creds [15:36:02] LGTM but social engineering warning :-P I would confirm via a videocall [15:37:31] arturo, can you review this MR? https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [15:37:41] chuckonwu: 👀 [15:39:43] chuckonwu: LGTM, +1d, please merge and applu [15:39:45] apply* [15:41:00] chuckonwu: we got the commit message wrong :-( https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/commit/5d08bfa012b75033702b92fcf770823a19305daf [15:41:16] I'll fix that myself by force-pushing. You will need to rebase your local copy [15:42:23] arturo: xd, I'm up for it if you want, I can confirm on telegram too [15:43:05] https://meet.google.com/zzi-wine-bdm [16:09:43] chuckonwu: look at https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/commits/main?ref_type=heads I have force pushed to replace the last commit [16:09:56] please rebase your local copy [16:13:17] * arturo offline