[08:47:34] <arturo>	 taavi: did you started tracking the dafult sg for toolsbeta in tofu-infra?
[08:49:19] <taavi>	 arturo: no, didn't have time for that yet
[08:49:30] <arturo>	 ok, np
[08:49:39] <arturo>	 I was just curious how that went
[12:54:18] * arturo now wants to see how many IPv4 vs IPv6 traffic is flowing in the network
[12:54:44] <taavi>	 i want one of those "Average IP version" grafana panels
[12:55:13] <taavi>	 arturo: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218 maybe something for monday?
[12:56:48] <taavi>	 or today, if you're feeling bold
[13:00:12] <arturo>	 should be safe for today, is just rules
[13:00:22] <arturo>	 (last words??)
[13:00:49] <arturo>	 also, reminds me of that little refactor to make that array a map instead
[13:01:23] <arturo>	 if we do before, it would be a smaller work in the state compared to after merging that patch
[13:01:42] <taavi>	 i'd rather not block this on that
[13:01:47] <arturo>	 anyway, I'm about to go for food, we can do either later
[13:02:19] <arturo>	 I can do the refactor myself later
[13:02:26] <topranks>	 maybe we should wait before building that graph:
[13:02:27] <topranks>	 https://w.wiki/DuVn
[13:02:30] <arturo>	 then merge this on monday?
[13:03:46] <taavi>	 arturo: sounds good
[13:04:02] <topranks>	 taavi: cloudlb2002-dev seems good now??
[13:04:23] <topranks>	 I guess next step is to check things work over v6, then probably publish some AAAA records?
[13:04:57] <taavi>	 topranks: https://phabricator.wikimedia.org/T379282#10767585
[13:05:32] <topranks>	 taavi: ah cool 
[13:05:38] <topranks>	 that's a very good point about the AAAA records 
[13:06:03] <topranks>	 an added complication as the hosts have v6 addr on the other interface 
[13:07:14] <taavi>	 yea, i recall we talked about how to best do that a few weeks back but i haven't paid any more thought onto that until now
[13:26:52] <andrewbogott>	 taavi: thank you for finishing up cloudlb2004-dev! I'm going to shut down 2001-dev today unless you want the honor
[13:28:06] <topranks>	 was the bgp added for cloudlb2004-dev?
[13:28:53] <taavi>	 no, I removed the session for 2001-dev to make the alerts go away but didn't add one for 2004-dev yet
[13:29:48] <topranks>	 taavi: ok cool, if you're happy to do that then great 
[13:29:57] <andrewbogott>	 Ok, we need that then (I saw that you had made a change but didn't realize it was a delete and not a move)
[13:30:02] <taavi>	 i'll do it now
[13:30:14] <andrewbogott>	 great!
[13:32:42] <taavi>	 172.20.5.5            64605          4          2       0       1          19 Establ
[13:34:06] <taavi>	 andrewbogott: cloudlb2004-dev needs a reboot to pick up the needed routing rules, I'll do that now
[13:34:38] <andrewbogott>	 sgtm
[13:35:26] <taavi>	 there's some bits of network config that puppet only adds to /e/n/i only, instead of also running the needed commands to enable them up immediately :/
[13:35:55] <andrewbogott>	 that's not great but I guess we don't change those every day
[13:40:39] <topranks>	 puppet meddling with e/n/i isn't great but we don't have a better option right now 
[13:41:14] <topranks>	 I think the bigger problem is ifupdown which doesn't do a good job of re-applying things in the correct order etc. if they are added after startup 
[13:43:06] <andrewbogott>	 taavi: I want to stop haproxy on cloudlb200[23] and watch it still work. Are we ready for that?
[13:44:06] <taavi>	 andrewbogott: yes. cloudlb2002-dev is the only one at the moment announcing the v6 address but that's not in use anywhere so its fine
[13:45:06] <andrewbogott>	 hm, 500
[13:45:15] <andrewbogott>	 which I guess means traffic as least getting there
[13:45:30] <taavi>	 at least https://keystone.openstack.codfw1dev.wikimediacloud.org/ still works
[13:46:26] <andrewbogott>	 It does?
[13:46:31] <andrewbogott>	 # openstack service list
[13:46:31] <andrewbogott>	 Internal Server Error (HTTP 500)
[13:47:31] <taavi>	 i see lots of errors in the keystone-admin service unit logs
[13:47:52] <andrewbogott>	 yeah, I'm guessing that the issue is with DB access
[13:48:09] <andrewbogott>	 well, wait, that doesn't go through haproxy does it?
[13:48:18] <taavi>	 it does
[13:48:54] <andrewbogott>	 I but I have to refresh the grants to allow access from the new haproxy
[13:48:56] <andrewbogott>	 let me do that...
[13:49:25] <taavi>	 also please re-enable the haproxies when you can, otherwise you're causing a bunch of scary-looking alerts on -operations
[13:50:28] <andrewbogott>	 I did that a while ago
[13:50:54] <taavi>	 bird and anycast-healthchecker are still down
[13:53:35] <andrewbogott>	 apparently they needed a puppet run to restart the
[13:53:48] <andrewbogott>	 but now I'm going to break everything again because I think 2004-dev will work now
[13:55:46] <andrewbogott>	 yeah, all working now
[13:58:56] <andrewbogott>	 thanks again taavi, I'm going to start the decom process and then get breakfast while it grinds
[16:16:53] <arturo>	 andrewbogott: you may want to double check why s3 @ codfw1dev doesn't seem to work
[16:17:31] <arturo>	 andrewbogott: nevermind, I'm seeing old data now
[16:22:20] <andrewbogott>	 it came back?
[16:22:27] <andrewbogott>	 I haven't looked since yesterday
[16:23:05] <andrewbogott>	 the horizon UI works but that's swift and not S3...
[16:25:15] <arturo>	 andrewbogott: nevermind, seems to work now
[16:25:45] <andrewbogott>	 now I sort wish it didn't
[16:26:13] <arturo>	 I was looking at an old pipeline failure here https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/pipelines
[16:26:26] <arturo>	 also the pipeline failure was a typo, not s3 failing
[16:27:09] * arturo off for the weekend
[16:27:14] * andrewbogott waves