[07:37:18] <taavi>	 arturo: when you have a moment, https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/6
[07:43:03] <arturo>	 taavi: LGTM +1
[07:43:44] <taavi>	 thanks, deploying
[07:48:53] <taavi>	 designate is returning 504s? https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/jobs/490149
[08:00:06] <taavi>	 i restarted things and it seems better
[08:00:09] <taavi>	 now running apply
[08:01:47] <taavi>	 toolserver.org now has an IPv6 address
[08:20:39] <arturo>	 🎉 
[08:24:05] <taavi>	 arturo: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/7
[08:39:09] <taavi>	 TIL that tools.wikimedia.de still exists (and is a CNAME to toolserver.org)
[08:39:19] <arturo>	 heh
[08:40:30] <taavi>	 anyway, any concerns about the newest MR or can I merge it?
[08:40:44] <arturo>	 I'm looking
[08:41:36] <arturo>	 I'm scratching my head regarding why the import block for records don't need the project_id
[08:41:48] <arturo>	 I guess is because is the same as the auth
[08:42:14] <arturo>	 anyway, approved +1
[08:45:00] <taavi>	 thanks
[08:47:14] <taavi>	 one last MR: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/8
[08:48:58] <arturo>	 👀
[08:52:08] <arturo>	 +1'd
[08:52:14] <arturo>	 (but there is a typo)
[08:52:26] <taavi>	 thanks, fixing and will merge then
[08:52:53] <taavi>	 btw I think we can close T380174 now?
[08:52:53] <stashbot>	 T380174: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174
[08:53:25] <arturo>	 yes
[08:55:02] <arturo>	 also I just closed the 13 years old T37947
[08:55:03] <stashbot>	 T37947: Enable IPv6 on CloudVPS - https://phabricator.wikimedia.org/T37947
[08:55:05] <taavi>	 there's also T37947, with a few open subtasks, and that one should probably be open until docs and such have been updated
[08:55:07] <taavi>	 ah
[08:55:24] <taavi>	 docs is T380054 apparently
[08:55:25] <stashbot>	 T380054: Cloud VPS: prepare  documentation on VXLAN/IPV6 migration - https://phabricator.wikimedia.org/T380054
[08:55:35] <taavi>	 i think T380081 is also fixed?
[08:55:35] <stashbot>	 T380081: horizon: enable the UI to select networks on VM creation panel - https://phabricator.wikimedia.org/T380081
[08:55:46] <arturo>	 yeah both of them are nearly done
[08:55:52] <arturo>	 just last few checks, then close them
[08:55:57] <taavi>	 cool
[09:52:54] <arturo>	 taavi: https://usercontent.irccloud-cdn.com/file/dbDK98Ql/image.png
[09:54:27] <taavi>	 arturo: i'll look, but suspect it's a monitoring issue as i just fixed T392559 to make it appear as a target in the first place
[09:54:28] <stashbot>	 T392559: metricsinfra: maintain-projects is broken - https://phabricator.wikimedia.org/T392559
[09:54:37] <arturo>	 ok
[09:55:39] <taavi>	 yup, metricsinfra is trying to scrape it over v6 from a v4-only instance :D
[09:57:00] <taavi>	 i need to get some food, will fix after that
[10:21:21] <arturo>	 topranks: I added this FQDN for this addr: https://netbox.wikimedia.org/ipam/ip-addresses/18719/ I hope the PTR zone has been set up already? :-S
[10:22:24] <taavi>	 arturo: did you run the cookbook to update the zone files?
[10:22:31] <arturo>	 $ sudo cookbook sre.dns.netbox 'neutron updates'
[10:22:38] <arturo>	 this one? ^^^ 
[10:22:40] <taavi>	 yes
[10:22:45] <arturo>	 is running
[10:22:52] <arturo>	 but that one fails if the zone is not set already?
[10:23:03] <taavi>	 it shouldn't
[10:23:11] <arturo>	 ok
[10:23:14] <taavi>	 it might not have any effect, but it won't crash i think
[10:23:51] <topranks>	 It's not the "zone" that doesn't exist as such, but the snippet that the Netbox DNS automation file will create is not referenced in the zone that does 
[10:24:05] <topranks>	 WMF DNS is auth for 0.8.c.e.2.0.a.2.ip6.arpa. 
[10:25:24] <arturo>	 it should work just fine
[10:25:26] <arturo>	 https://www.irccloud.com/pastebin/BWIRKY8A/
[10:26:11] <topranks>	 nah 
[10:26:42] <topranks>	 oh wait 
[10:26:46] <topranks>	 sorry - yeah I did it already 
[10:27:01] <topranks>	 there is a record for the cloudgw IP 
[10:27:01] <topranks>	 https://netbox.wikimedia.org/ipam/prefixes/1100/ip-addresses/
[10:27:21] <topranks>	 and the required "INCLUDE" statement is in the 0.8.c.e.2.0.a.2.ip6.arpa zone file:
[10:27:31] <topranks>	 cmooney@dns2005:/etc/gdnsd/zones$ grep 4.0.e.f.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa * 
[10:27:31] <topranks>	 0.8.c.e.2.0.a.2.ip6.arpa:$INCLUDE netbox/4.0.e.f.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa
[10:27:35] <topranks>	 so yeah should be fine 
[10:28:02] <arturo>	 thanks for double checking
[10:29:19] <topranks>	 but anyway please do check this before adding the stuff in Netbox.  If it's missing in the dns it causes problems, and blocks SREs doing reimages and other stuff that trigger the dns cookbook 
[10:29:35] <arturo>	 yes, you are right
[10:30:21] <topranks>	 if it's a new range this is the kind of thing you need to add:
[10:30:21] <topranks>	 https://gerrit.wikimedia.org/r/c/operations/dns/+/1109732
[10:30:35] <topranks>	 the process is quite crunky, I'm hoping to push for something saner in time we don't need to do it this way 
[10:31:02] <arturo>	 👍
[10:43:39] <taavi>	 arturo: filed T392570 for the prometheus issue, unfortunately it seems a bit complex
[10:43:40] <stashbot>	 T392570: metricsinfra: Support scraping v6-enabled instances - https://phabricator.wikimedia.org/T392570
[10:44:59] <arturo>	 taavi: thanks! all the options you listed seems sensible to me
[10:59:39] <arturo>	 functional network tests in eqiad1:
[10:59:41] <arturo>	 [2025-04-24 10:50:25] INFO: --- passed tests: 78
[10:59:41] <arturo>	 [2025-04-24 10:50:25] INFO: --- failed tests: 0
[10:59:41] <arturo>	 [2025-04-24 10:50:25] INFO: --- total tests: 78
[11:22:38] <paladox>	 hi, is it expected that tools.wmflabs.org Ipv6 doesn't work?
[11:24:08] <taavi>	 paladox: if you mean pinging it doesn't work, it seems like we're missing a security group rule
[11:24:16] <taavi>	 i'll have a look once i'm out of meetings
[11:24:22] <paladox>	 ah ok, thanks!
[11:24:28] <taavi>	 but i think http(s) itself should work?
[11:25:35] <paladox>	 yup, that works
[11:26:04] <topranks>	 nice :) 
[11:26:09] <topranks>	 cmooney@wikilap:~$ curl -v https://tools.wmflabs.org
[11:26:09] <topranks>	 *   Trying 2a02:ec80:a000:1::304:443...
[11:26:09] <topranks>	 * Connected to tools.wmflabs.org (2a02:ec80:a000:1::304) port 443 (#0)
[11:43:11] <taavi>	 arturo: seemingly the default security group has a allow all ICMP v4 rule. any concerns about duplicating that as is for ipv6? generally that means that all v6 instances will be pingable from the outside (compared to v4 instances where that only applied to those with floating IPs), but I think that's fine
[11:45:26] <topranks>	 My general instincts are to allow all ICMP 
[11:45:56] <topranks>	 one could limit to only ICMP ECHO allowed in from outside, along with RELATED, ESTABLISHED ICMP (i.e. packet too big messages and other necessary things) 
[11:47:05] <topranks>	 You also need to make sure not to break neighbor discovery, router advertisements and the like, that would mess everything up (though it's only local to the link / LAN) 
[11:50:56] <arturo>	 taavi: yeah, seems fine
[11:52:04] <taavi>	 i added those by hand to tools / toolsbeta
[11:52:12] <taavi>	 we should maybe find a way to backfill those to all existing projects
[11:52:26] <taavi>	 shame that neutron can't do a single rule that matches both protocols :/
[11:52:29] <arturo>	 taavi: tofu-infra tracks that
[11:52:33] <topranks>	 confirm it's working 
[11:52:36] <topranks>	 cmooney@wikilap:~$ ping tools.wmflabs.org  
[11:52:36] <topranks>	 PING tools.wmflabs.org(tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud (2a02:ec80:a000:1::304)) 56 data bytes
[11:52:36] <topranks>	 64 bytes from tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud (2a02:ec80:a000:1::304): icmp_seq=1 ttl=52 time=89.2 ms
[11:52:36] <topranks>	 64 bytes from tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud (2a02:ec80:a000:1::304): icmp_seq=2 ttl=52 time=88.9 ms
[11:52:56] <taavi>	 arturo: I see https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/blob/main/modules/project/default_sg_rules.tf?ref_type=heads#L69 but that definitely was not applied on tools, so I think that's for new projects only?
[11:53:28] <arturo>	 well, tools is definitely interesting: do we want to track that in tofu-infra or in the dedicated tofu-provisioning repo?
[11:54:05] <taavi>	 i think the default security group rules should be in tofu-infra to not duplicate them, and anything custom in the toolforge repo
[11:54:15] <arturo>	 ok
[11:54:31] <arturo>	 then the way to go is to drop all the rules by hand at the time of tofu apply
[11:54:42] <arturo>	 there is a brief moment in which there are no rules, but that's generally OK
[11:55:35] <taavi>	 I see a "manage_default_secgroup = false," setting in resources/eqiad1/tools/main.tf, I think that's the one that we need to flip
[11:55:43] <arturo>	 yes
[11:56:04] <taavi>	 can we import the rules instead? not having the rules to allow project-internal traffic there even for a second is very scary
[11:56:25] <arturo>	 import rules is tedious, I don't recommend
[11:56:32] <taavi>	 ok
[11:56:34] <arturo>	 most of the traffic is covered by other specific security groups
[11:56:42] <taavi>	 will neutron let us add duplicate rules and remove the old ones afterward?
[11:56:46] <arturo>	 no
[11:56:47] <arturo>	 :-(
[11:56:50] <taavi>	 aha
[11:57:37] <arturo>	 most of connections are established already, so if a rule is dropped, the connection wont break anyway
[11:57:48] <taavi>	 true
[11:58:13] <arturo>	 the brief block only affects new connection (theory)
[11:58:32] <taavi>	 ok, let's try it in toolsbeta?
[11:58:37] <arturo>	 for the projects I backfilled, I did not notice a cut
[11:58:39] <arturo>	 sure
[11:59:14] <arturo>	 more info: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/tofu-infra#Managing_the_default_security_group
[12:04:27] <arturo>	 taavi: I guess this is not a priv key, right? https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3/diffs#8ca01d86f9efb98b375fde98a41614a36938a3a8_4_55
[12:04:45] <taavi>	 arturo: no, that's the public key
[12:04:53] <arturo>	 thanks for confirming
[12:06:16] <arturo>	 chuckonwu: do you think https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/3 is ready to merge?
[12:07:41] <taavi>	 hmm, I do not think the floating IP records should be imported there, they're already automatically managed via a script in puppet
[12:08:26] <arturo>	 the ones with 'MANAGED BY dns-floating-ip-updater.py IN PUPPET' 
[12:08:28] <arturo>	 ?
[12:08:29] <taavi>	 yes
[12:08:38] <arturo>	 makes sense
[12:13:14] <arturo>	 taavi: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/9 does this make sense?
[12:13:49] <taavi>	 lgtm
[12:16:18] <arturo>	 s3 in codfw1dev is failing?
[12:16:21] <arturo>	 https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/jobs/490349
[12:29:59] <taavi>	 I plan to decline this: T209011 in favour of IPv6
[12:29:59] <stashbot>	 T209011: Change routing to ensure that traffic originating from Cloud VPS is seen as non-private IPs by Wikimedia wikis - https://phabricator.wikimedia.org/T209011
[12:30:25] <arturo>	 sounds good
[12:35:07] <taavi>	 arturo: https://wikitech.wikimedia.org/wiki/News/2025_Cloud_VPS_VXLAN_IPv6_migration#Timeline why not make dualstack the default?
[12:35:54] <taavi>	 i also would make either of the VXLAN options the default today, instead of waiting two months to change that
[12:36:22] <arturo>	 maybe we can discuss later in the weekly team meeting. I wanted to be extra carefull
[13:19:41] <taavi>	 i have added a general plan to T379175
[13:19:41] <stashbot>	 T379175: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175
[13:24:31] <dhinus>	 hmm I just got an error running tofu plan (from the cookbook)
[13:24:33] <dhinus>	 Error: Failed to get existing workspaces: operation error S3: ListObjectsV2, exceeded maximum number of attempts, 5, https response error StatusCode: 503
[13:24:52] <dhinus>	 that was for codfw, during "Initializing modules..."
[13:25:39] <dhinus>	 actually I think it's during "Initializing the backend...", I can reproduce
[13:26:28] <dhinus>	 just saw arturo's message from 1 hour ago: "s3 in codfw1dev is failing?"
[13:27:35] <dhinus>	 can I get a +1 anyway for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/202
[13:27:44] <dhinus>	 I will apply to eqiad1 only
[13:28:19] <taavi>	 +1
[13:28:34] <dhinus>	 thanks
[13:33:41] <chuckonwu>	 arturo, yes that branch is ready for merging
[14:07:12] <arturo>	 chuckonwu: approved, please hit the merge button and then run the apply pipeline step :-)
[14:08:36] <taavi>	 i've just now realized https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/blob/main/toolsbeta/dns.tf?ref_type=heads#L13 is something managed by the web proxy service, not sure if that should be in tofu code as well
[14:09:13] <arturo>	 ah, good point
[14:09:40] * arturo now wonders how to interact with the tofu state via a cli
[14:10:10] <chuckonwu>	 I haven't yet run the apply stage so this is a quick patch, one second
[14:10:22] <arturo>	 oh good
[14:13:22] <taavi>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1138799 would add a description to any future web proxy records to avoid that confusion
[14:14:00] <arturo>	 +1'd
[14:15:26] <taavi>	 andrewbogott: I'm going to rebase and merge your dynamicproxy puppet changes
[14:18:37] <arturo>	 chuckonwu: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/10 +1'd
[14:18:39] <chuckonwu>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/10 removes the record that is managed by Puppet
[14:18:42] <chuckonwu>	 Great
[14:20:43] <arturo>	 thanks!
[14:40:51] <arturo>	 taavi: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/18 ?
[14:40:55] <arturo>	 wdyt?
[14:41:27] <taavi>	 who does root@wmcloud.org go to?
[14:41:39] <arturo>	 us, I hope.
[14:43:08] <arturo>	 do you have other suggestions for the dest addr?
[14:43:43] <taavi>	 cloud-admin-feed@lists?
[14:44:11] <arturo>	 k
[14:44:12] <taavi>	 does sendmail really just work like that from a ci job? huh
[14:44:31] <arturo>	 I was _hoping_ 
[14:44:33] <arturo>	 🙏
[14:44:42] <taavi>	 we can give it a try
[14:46:41] <taavi>	 that's failing in codfw1dev for some reason
[14:46:53] <arturo>	 yeah, can't somehow find the s3 bucket
[14:47:02] <arturo>	 also affects tofu-infra
[14:47:26] <arturo>	 may be related to the ceph work andrewbogott is doing on codfw1dev
[14:47:44] <taavi>	 +1
[14:48:44] <dcaro>	 oh, I saw the other day that tofu is adding native locking for s3 bucket state
[14:48:51] <dcaro>	 (something we discussed at some point)
[14:49:07] <arturo>	 if this email thing work, I will add it next to the toolforge repo
[14:50:36] <arturo>	 dcaro: nice!
[14:51:25] <dcaro>	 https://github.com/opentofu/opentofu/commit/eba25e2fedca0a782bdf9d381a3a65f3ad1acff3
[14:55:31] <topranks>	 taavi: one thing I notice 
[14:55:55] <topranks>	 on this page:  https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_IP_space
[14:56:08] <topranks>	 it seems to only list the eqiad ranges?  
[14:56:26] <topranks>	 if that's the intent I think the v4 private range should be 172.16.0.0/17 as that is all that is routed to the cloudgw's 
[14:56:38] <topranks>	 and we have space from the upper half of the /16 used in codfw 
[14:57:16] <taavi>	 topranks: yeah, i was thinking about that.. in my mind that page is for our users for problems like "what IPs should I set the access list to in my on-wiki credentials and such", which is why that doesn't include codfw1dev with no real user workloads
[14:57:27] <taavi>	 i will update it to only have the first /17. thanks
[14:57:36] <topranks>	 yeah no worries, not a big deal 
[14:57:54] <topranks>	 the other alternative would be to include codfw1dev ranges, so we'd need to add the v6 range for there 
[14:58:16] <topranks>	 but as you say codfw not used by user workloads so I think it makes more sense to change the v4 mask 
[14:58:29] <taavi>	 https://wikitech.wikimedia.org/w/index.php?title=Help:Cloud_VPS_IP_space&curid=458702&diff=2295736&oldid=2295042
[14:58:42] <topranks>	 cool <3
[15:42:52] <arturo>	 email proposal:
[15:42:57] <arturo>	 https://www.irccloud.com/pastebin/sr5L7tkF/
[15:43:58] <taavi>	 arturo: s|make sure VXLAN/IPv6-dualstack is|make sure the VXLAN/IPv6-dualstack network is|
[15:44:06] <taavi>	 can you update the timeline on the wikitech page as well?
[15:44:11] <arturo>	 yes
[15:46:09] <arturo>	 taavi: https://wikitech.wikimedia.org/w/index.php?title=News%2F2025_Cloud_VPS_VXLAN_IPv6_migration&diff=2295759&oldid=2295688
[15:47:08] <taavi>	 maybe push back disabling VLAN/legacy a bit further?
[15:47:10] <taavi>	 otherwise lgtm
[15:47:42] <arturo>	 ok, 2 months
[15:47:51] <taavi>	 wfm
[15:48:13] <taavi>	 topranks: the v6 BGP session for cloudlb2002-dev should now be using the correct interface, if you want to set the switch-side config
[15:49:25] <taavi>	 arturo: it still has the default switch (in a week) and disabling legacy creation in one item?
[15:49:39] <taavi>	 i'll fix
[15:50:16] <bd808>	 Should it be possible for me to ping6/traceroute6/curl -6 toolserver.org at this point?
[15:50:20] <arturo>	 taavi: yes, please fix it. I have my child demanding my attention now 🌈
[15:50:30] <taavi>	 bd808: yes
[15:50:47] <bd808>	 taavi: I have sad news then. All 3 fail.
[15:50:59] <bd808>	 I'll make a paste
[15:56:47] * arturo offline
[15:57:11] <taavi>	 huh
[15:57:11] <taavi>	 Apr 23 15:33:55 tools-legacy-redirector-3 systemd-networkd[455]: ens3: DHCPv6 address 2a02:ec80:a000:1::304/128 (valid for 23h 59min 59s, preferred for 23h 59min 59s)
[15:57:11] <taavi>	 Apr 24 15:33:55 tools-legacy-redirector-3 systemd-networkd[455]: ens3: DHCPv6 lease lost
[15:58:20] <bd808>	 taavi: https://phabricator.wikimedia.org/P75450 -- there isn't a lot of information there, but let me know if I can check something more useful.
[16:05:19] <taavi>	 aha, the host-level firewall is blocking dhcpv6 response packets for some reason
[16:07:36] <taavi>	 bd808: tracking this in T392611 now
[16:07:37] <stashbot>	 T392611: VMs with ferm host-level firewall do not permit DHCPv6 responses - https://phabricator.wikimedia.org/T392611
[16:08:16] <bd808>	 :old-man-shakes-fist-at-ferm:
[16:16:23] <topranks>	 this also happens in iptables and nftables I believe 
[16:17:23] <topranks>	 at least from my home connection doing DHCPv6 IA_PD, the request is sent to a multicast address, reply comes from a unicast one, and thus doesn't match "related, established" 
[16:17:38] <topranks>	 I need this instead:
[16:17:39] <topranks>	         iifname "enp2s0" ip6 saddr fe80::/10 udp dport 546 accept
[16:18:04] <taavi>	 topranks: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1138837
[16:18:18] <topranks>	 taavi: hey was afk just back 
[16:18:26] <topranks>	 let me look at that and the bgp thing you mentioned 
[16:18:32] <taavi>	 yeah no hurry for the bgp thing
[16:19:13] <taavi>	 PCC at https://puppet-compiler.wmflabs.org/output/1138837/5353/, for both an instance running a firewall and one not
[16:19:16] <topranks>	 +1 for the patch makes sense 
[16:19:29] <taavi>	 thanks!
[16:21:50] <taavi>	 bd808: try now?
[16:29:12] <bd808>	 taavi: the curl works! :)
[16:34:19] <topranks>	 nice :) 
[16:34:26] <topranks>	 taavI: when you get a minute
[16:34:27] <topranks>	 https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1138850
[17:22:41] <topranks>	 taavi: ok I fixed a few things and the BGP is up and working for the 2a02:ec80:a100:4000::1/128 vip from cloudlb2002 
[17:23:02] <topranks>	 well, for some value of "working" 
[17:23:16] <topranks>	 we are missing the required "ip rules" on cloudlb2002 to make it work 
[17:23:31] <topranks>	 right now it is getting packets but sending the response out the prod realm primary interface, following the default 
[17:24:25] <topranks>	 we need the equivalent of this for IPv6:
[17:24:31] <topranks>	 https://www.irccloud.com/pastebin/nwxPVK4i/
[17:24:50] <topranks>	 The IPv6 route is in the "cloud-private" table so we are good there:
[17:24:54] <topranks>	 cmooney@cloudlb2002-dev:~$ ip -6 route show table cloud-private 
[17:24:54] <topranks>	 default via 2a02:ec80:a100:205::1 dev vlan2151 metric 1024 pref medium
[17:25:57] <topranks>	 we just need to tell the system to use this table for packets from... well that's not 100% clear 
[17:26:19] <topranks>	 certainly 2a02:ec80:a100:4000::/64, but perhaps it should be 2a02:ec80:a100::/64 
[17:26:31] <topranks>	 sry.... 2a02:ec80:a100::/48 
[17:26:52] <andrewbogott>	 topranks: while you're in there, is cloudlb2004-dev also set up properly? I'm flipping service from 2001-dev to 2004-dev and assume some network things need doing. (2004-dev doesn't actually work yet but that's another story)
[17:28:13] <andrewbogott>	 I mean, haproxy doesn't work, the host itself should be fine.
[17:29:34] <topranks>	 andrewbogott: no bgp is not set up for it on the cloudsw
[17:29:40] <topranks>	 https://www.irccloud.com/pastebin/WjaSwuEY/
[17:30:45] <andrewbogott>	 Is it simple to switch service that from 2001 to 2004?
[17:30:54] <dcaro>	 the alerts on cloudcephosd1029 is me, not sure why they did not work (the cookbook did create a silence :/ )
[17:31:04] <dcaro>	 *the silences did not work
[17:31:07] <andrewbogott>	 oh good, thanks dcaro! I was just starting to worry
[17:31:24] <andrewbogott>	 and also briefly forgot that that server is the lab rab
[17:31:26] <topranks>	 andrewbogott: they can both be configured side by side 
[17:31:37] <topranks>	 however the bigger issue is bird is not running on 2004 
[17:31:42] <topranks>	 cmooney@cloudlb2004-dev:~$ sudo birdc
[17:31:42] <topranks>	 Unable to connect to server control socket (/run/bird/bird.ctl): No such file or directory
[17:31:55] <topranks>	 let's get that working before we trigger alerts by adding it on the cloudsw side and failing 
[17:32:01] <dcaro>	 andrewbogott: fyi there's a tmux running in cloudcumin1001 with the cookbook running
[17:32:13] <andrewbogott>	 yeah, that's because haproxy is crashing I think. And haproxy is crashing because the conf syntax is super different in Bookworm and I haven't updated things yet.
[17:32:24] <topranks>	 ok yes that could be it 
[17:32:28] <andrewbogott>	 dcaro: meaning I left it running? Or meaning you're leaving it running?
[17:32:41] <dcaro>	 nonon, I left it running xd
[17:32:45] <andrewbogott>	 ok :)
[17:32:55] <andrewbogott>	 topranks: this is https://phabricator.wikimedia.org/T392366 if you need a ref
[17:33:24] <andrewbogott>	 2001-dev is already mostly out of service so I don't mind if you break it in bgp
[17:33:28] <topranks>	 andrewbogott: sorry maybe we are talking past each other 
[17:33:33] <topranks>	 I was looking at cloudlb2004-dev 
[17:33:36] <topranks>	 you are working on that?
[17:33:43] <topranks>	 or cloudcephosd2004-dev ?
[17:33:58] <dcaro>	 andrewbogott: I'll leave it undraining in batches, in case something goes awry feel free to take it stop it
[17:34:41] * dhinus offline
[17:34:45] <andrewbogott>	 dcaro: this is putting 1029 back into the pool?
[17:34:57] <dcaro>	 yep
[17:35:00] <andrewbogott>	 topranks: shit, wrong task, hang on
[17:35:33] <andrewbogott>	 T377126
[17:35:34] <stashbot>	 T377126: replace cloudlb2001-dev with cloudlb2004-dev - https://phabricator.wikimedia.org/T377126
[17:35:36] <andrewbogott>	 there we go
[17:35:49] <topranks>	 ok yeah well that does make more sense 
[17:35:54] <topranks>	 and I know they run haproxy :) 
[17:36:37] <andrewbogott>	 ideally they do
[17:36:49] <topranks>	 ha yeah 
[17:36:56] <topranks>	 let me know how you get on with that
[17:37:03] <topranks>	 when it's done hopefully bird will be happy 
[17:37:14] <topranks>	 we can tell by running "sudo birdc" and then "show protocols" 
[17:37:27] <topranks>	 after which I'll add the session on the switch and we can see how it goes 
[17:37:35] <andrewbogott>	 ok
[17:40:39] <dcaro>	 oh, quick question, the support assist file for the hard drive replacement is too big to upload to phabricator, any ideas where can I put it?
[17:44:14] <andrewbogott>	 s3?
[17:44:14] <taavi>	 topranks: huh, I thought I got those rules added already. anyway, will have a look tomorrow
[17:44:17] <andrewbogott>	 Etherpad?
[17:44:21] <topranks>	 dcaro: google drive?
[17:44:21] <taavi>	 dcaro: people.wikimedia.org?
[17:44:26] <andrewbogott>	 dcaro: you're only going to get bad suggestions from me
[17:44:38] <taavi>	 depends on if that's needs to be private or not
[17:44:56] <andrewbogott>	 it probably does, and google drive is probably the right answer
[17:44:57] <dcaro>	 I think it does not, the disk procurement tickets are not private right?
[17:45:21] <andrewbogott>	 Basically anything to do with procurement or negotiation should be kept private if possible.
[17:45:35] <dcaro>	 I mean replacement sorry
[17:45:35] <taavi>	 andrewbogott: are you planning to update https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/2?
[17:45:38] <dcaro>	 (same same I guess)
[17:46:09] <andrewbogott>	 taavi: yes but also my day is 100% out of control so don't save it for me if you want it done.
[17:46:27] <dcaro>	 taavi: thanks for the link, I did not know about that
[17:50:17] <andrewbogott>	 Can I reset the tofu-infra repo in cloudcontrol1011 or does someone have work in progress there?  (puppet is complaining)
[17:58:20] <dcaro>	 added the people.wikimedia.org to my wm-lol xd https://wm-lol.toolforge.org/api/v1/search?query=help
[17:58:27] * dcaro off
[18:21:27] * andrewbogott does the reset
[23:58:30] <bd808>	 "I have faith that we will get to IPv6 before the heat death of the universe, but I'm not willing to put a more definite date on it than that." -- bd808, https://phabricator.wikimedia.org/T209011#4734161