[10:07:17] <blancadesal>	 arturo: if you have a moment today, I'd appreciate if you could take a look at https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/8
[10:07:28] <arturo>	 sure
[10:07:58] <blancadesal>	 thanks
[10:15:48] <_joe_>	 hi, who can I ask for a review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1056937/ ?
[10:20:34] <arturo>	 _joe_: I can do it
[10:26:54] <arturo>	 cc dhinus 
[10:27:40] <dhinus>	 looking
[10:29:12] <dhinus>	 should be good, let's check the PCC
[10:31:15] <dhinus>	 pcc failed with 'no parameter named 'target''
[10:39:37] <dcaro>	 dhinus: do you know if the openstack config used in cloudcumin for cumin should work from my laptop? (or if it's using some special internal-only kind of endpoint)
[10:44:44] <dcaro>	 I'm getting unauthorized when trying to use novaobserver creds locally to list tools vms
[10:45:22] <dcaro>	 maybe some clouds.yaml stuff somewhere?
[10:48:58] <dhinus>	 hmm I haven't used novaobserver recently, so I don't know if something changed
[10:49:04] <dhinus>	 when was the last time you used it successfully?
[10:49:38] <dcaro>	 from cloudcumin1001 it works ok
[10:52:36] <dcaro>	 the auth looks ok to me locally, it reads the config ok
[10:52:45] <dcaro>	 but ends up in 401
[10:52:46] <dcaro>	 https://www.irccloud.com/pastebin/Vy5UCrst/
[10:53:02] <dcaro>	 (putting a pdb line in the middle of the openstack backend code for cumin)
[10:53:49] <dhinus>	 maybe it generates a token correctly, but then that token is rejected by the openstack service?
[10:53:50] <dcaro>	 the alternative is hardcoding the tools nodes in the cookbooks (of which we have some already, so might not be too bad for now, then we change all when it works)
[10:54:16] <dcaro>	 might be
[10:54:58] <dhinus>	 same cookbook from cloudcumin works? that is also using novaobserver
[10:55:07] <dhinus>	 (/etc/cumin/config.yaml)
[10:55:33] <dhinus>	 clouds.yaml should not be involved if you go through spicerack
[10:55:48] <dcaro>	 yep
[10:55:52] <dcaro>	 it works yes
[10:56:15] <dhinus>	 is your local cumin config file similar to the one in cloudcumin?
[10:57:00] <dcaro>	 yep, quite similar yes, I copy-pasted the openstack part of it
[10:57:14] <dhinus>	 I'm actually confused why the creds are in the cumin config and not in the spicerack config
[10:57:20] <dcaro>	 I think I might have messed up the virtualenv when installing the openstack packages :/
[10:57:20] <dhinus>	 ah I know
[10:57:24] <dhinus>	 to query the server list
[10:57:47] <dhinus>	 it's possible it's not reading the config file you expect it to read
[10:57:53] <dcaro>	 now cookbook itself is not running (bad numpy version or something)
[10:58:00] <dcaro>	 I'll try to fix that and see if that helps
[10:58:22] <dcaro>	 it's using the right config file, I can pdb in the middle of the cumin code, and see that it read the right config (user/pass/url/...)
[10:59:15] <dcaro>	 it seems cookbook only supports numpy<2
[11:04:30] <dcaro>	 `dcaro@urcuchillay$ wmcs-cookbooks wmcs.toolforge.k8s.component.deploy --cluster-name toolsbeta --component builds-cli --git-branch bump_to_0.0.18`
[11:04:33] <dcaro>	 that almos works :)
[11:13:59] <dcaro>	 \o/
[11:14:01] <dcaro>	 https://www.irccloud.com/pastebin/mtLzXrbF/
[11:14:53] <arturo>	 did you consider having a different cookbook?
[11:29:07] <dcaro>	 yes, but I don't want to have to think which cookbook to use depending on the component I want to deploy
[11:29:46] <dcaro>	 I'm also considering dropping `k8s` from the prefix, as it's not needed anymore
[11:35:22] <dcaro>	 same for the ToolforgeK8s* classes  
[11:35:25] <dcaro>	 and inventory
[11:41:57] <arturo>	 I keep being surprised by the flakiness of openstack
[11:42:05] <arturo>	 T371242
[11:42:05] <stashbot>	 T371242: openstack: codfw1dev: nova-compute can't contact rabbitmq - https://phabricator.wikimedia.org/T371242
[12:30:31] <dcaro>	 ready for review :), deploying also packages now https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1057847
[12:47:07] <arturo>	 andrewbogott: when you are awake, I would appreciate your input here: T371242
[12:47:10] <stashbot>	 T371242: openstack: codfw1dev: nova-compute can't contact rabbitmq - https://phabricator.wikimedia.org/T371242
[13:19:41] <dcaro>	 arturo: is there connectivity from the cloudcontrol to rabbitmk?
[13:19:45] <dcaro>	 *rabbitmq
[13:26:42] <dcaro>	 I think the new cloudcontrols might be missing the private vlan
[13:27:12] <dcaro>	 2151 interface is missing on 2009-dev
[13:27:35] <dcaro>	 the error is in 2005 though ,looking
[13:28:13] <dcaro>	 2005 can connect
[13:28:18] <dcaro>	 https://www.irccloud.com/pastebin/DV5fDenv/
[13:41:21] <andrewbogott>	 arturo, if you look in nova.conf for 'ALERT' you may find something useful :)
[13:43:06] <andrewbogott>	 (that's meaningful if the rabbit IPs changed but not if the control nodes changed)
[14:02:39] <andrewbogott>	 dcaro: I'm going to rebuild the harbor DB in a moment.
[14:04:59] <andrewbogott>	 done :)
[14:06:17] <dcaro>	 that was quick :)
[14:06:53] <andrewbogott>	 yeah, it turns out to be pretty reliable once I fixed the postgres rebuild path
[14:07:04] <dcaro>	 things look ok
[14:07:23] <andrewbogott>	 cool
[14:17:27] <arturo>	 why do you think there was any change to rabbitmq?
[14:18:03] <arturo>	 andrewbogott: I haven't changed anything related to rabbitmq or the network
[14:18:14] <andrewbogott>	 arturo: I don't think I have the context, what were you doing when things broke?
[14:18:44] <arturo>	 I updated python3-nova on all codfw1dev servers because https://phabricator.wikimedia.org/T371240
[14:19:25] <andrewbogott>	 hm
[14:19:37] <andrewbogott>	 then there's no reason why it should've broken :(  Do you know for sure that it was working pre-upgrade?
[14:19:46] <arturo>	 no :-(
[14:20:00] <arturo>	 I suspect it wasn't working before
[14:20:16] <arturo>	 but I only looked for nova-fullstack after the upgrade
[14:20:28] <arturo>	 well, maybe there are earlier logs
[14:21:07] <andrewbogott>	 hm, I guess now that the fullstack tests delete the oldest broken VMs we can no longer use that to answer this question...
[14:23:37] <arturo>	 I see plenty of errors in the logs from days before
[14:23:40] <andrewbogott>	 did you do a full restart of nova services recently?  I just restarted on a cloudvirt and that seems to have made it h appy
[14:23:45] <arturo>	 so this has nothing to do with the upgrade
[14:24:12] <arturo>	 is not happy, nova-compute will timeout in a bit waiting for rabbitmq
[14:24:38] <arturo>	 I have restarted all nova services using the cookbook this morning
[14:24:41] <andrewbogott>	 ok.  do you mind if I do 'sudo cookbook wmcs.openstack.restart_openstack --all --cluster-name codfw1dev' ?
[14:24:51] <arturo>	 sure
[14:25:02] <arturo>	 I did the same but with --nova
[14:25:07] <andrewbogott>	 (assuming it will produce the same result as when you did it)
[14:25:30] <arturo>	 that cookbook doesn't restart rabbitmq BTW
[14:25:37] <arturo>	 maybe we should have an option
[14:25:45] <arturo>	 but I did restart rabbitmq manually anyway
[14:27:57] <andrewbogott>	 the message we're seeing ('timed out waiting for reply') doesn't necessarily mean that nova can't talk to rabbit does it?
[14:28:14] <andrewbogott>	 Couldn't it mean that it can't talk to another service it's trying to talk to /via/ rabbit?  e.g. another nova agent, or neutron?
[14:28:15] <arturo>	 I think network-wise, as in TCP/IP we are fine
[14:29:16] <arturo>	 I have visually scanned all services logs using logstash, and I have found nothing relevant so far
[14:29:32] <andrewbogott>	 but also in terms of rabbit connection handshake
[14:30:57] <arturo>	 It is hard to navigate logs anyway, we are generating 2.5M log lines per day in codfw1dev ...
[15:40:33] <arturo>	 andrewbogott: I also just created T371274
[15:40:33] <stashbot>	 T371274: openstack: codfw1dev: manually clean up old hypervisors - https://phabricator.wikimedia.org/T371274
[15:41:28] <andrewbogott>	 thx. at the moment nova is convinced that there are still valuable VMs on 2001/2002 even though there aren't
[15:42:08] * arturo offline
[17:33:44] * dcaro off
[20:05:11] <bd808>	 andrewbogott: I have a task in need of prod root help at T371289. The credentials that Striker uses to talk to GitLab are expiring and I need someone with similar access rights as you to rotate the config value for me. Details of where to find the new secret are in the task.
[20:05:12] <stashbot>	 T371289: Rotate StrikerBot GitLab PAT before it expires on 2024-08-01 - https://phabricator.wikimedia.org/T371289
[20:15:45] <andrewbogott>	 I will give it a try! Might be a bit before I can look though, trying to get the damn AC to work
[20:24:10] <bd808>	 andrewbogott: it can totally wait until tomorrow. Good luck with local temperature controls
[20:45:24] <andrewbogott>	 bd808: I see hieradata/role/eqiad/wmcs/openstack/eqiad1/cloudweb.yaml but not hieradata/role/eqiad/wmcs/openstack/eqiad1/labweb.yaml -- cloudweb is the right place, right?
[20:48:00] <andrewbogott>	 ah, yep, apparently I renamed it :)
[20:55:36] <bd808>	 andrewbogott: I didn't pay much attention to the bits that I was cut-and-pasting from old emails. :) Glad you figured out the rename.
[20:57:41] <andrewbogott>	 Can you confirm that the key actually updated in the place it needed to update?
[20:58:15] * bd808 logs into a cloudweb
[20:59:49] <bd808>	 andrewbogott: cloudweb1003:/etc/striker/env still has the old value. Should I force a puppet run there?
[21:00:01] <andrewbogott>	 yep
[21:00:13] <andrewbogott>	 (sorry, still not 100% tuned in)
[21:01:20] <bd808>	 yup, it is updated now.
[21:01:23] <andrewbogott>	 great
[21:03:50] <bd808>	 GitLab decided that if you don't pay them you can only have a PAT that lasts for 365 days so we will get to do this dance once a year now
[21:12:42] <andrewbogott>	 Better 365 than 30 :(
[21:27:09] <bd808>	 In some ways 30 would be better because it would push us to find a fix for either automated rotation or removing the ridiculous constraint on the TTL.