[07:23:08] greetings [09:22:07] morning [09:22:35] o/ [11:17:03] andrewbogott: I think that now that we have back the APIs as usual you can revert what was done for https://phabricator.wikimedia.org/T442554 [11:22:36] komla: do you have the list of projects to be deleted today due to the bullseye deprecation? [12:43:04] does someone know what is up with the failing harbor metrics on toolsbeta? [13:49:15] reverting the secgroupadmin creds for quarry: https://github.com/toolforge/quarry/pull/109 and for paws: https://github.com/toolforge/paws/pull/541 [14:09:11] I'm trying to deploy paws but it's failing with "Error creating OpenStack container infra client: Resource not found: [POST https://openstack.eqiad1.wikimediacloud.org:25000/v3/auth/tokens]" [14:09:20] andrewbogott: this is likely related to https://github.com/toolforge/paws/commit/873f876541bf8f41ce10076c1b5a0b45afdd37e7 [14:09:53] wow, I definitely didn't think someone would redeploy during the hour that that patch is in flight, sorry! [14:09:57] If we merge it should resolve things for you. [14:10:18] I think that one is merged, but says "should probably be reverted"? [14:10:27] ah yes found the new one [14:10:33] https://github.com/toolforge/paws/pull/541 [14:10:40] I can wait :) [14:10:46] I think I... linked to that in the comment immediately before your comment :) [14:11:02] if you want to +1 both of those then I'll merge right away [14:11:30] nice, sorry I read "for quarry" and thought it was unrelated... should've read to the end of the line:D [14:12:41] +1d both [14:12:49] thx [14:13:45] deploy should work now if you rebase [14:14:04] * dhinus tries [14:14:37] it works! thank you [14:14:49] great! [14:15:08] Now I need to delete those old creds from keystone, let's see if I get the right ones [16:05:03] hmm quarry seems unhappy, it's loading but very slow, and I see the alert flapping in -cloud-feed [16:09:01] definitely something happening https://grafana.wmcloud.org/d/85040a6f-6ede-4970-aaa4-53d4becb7f12/quarry-overview?from=now-7d&to=now&timezone=utc [16:10:05] seems like it might not be directly traffic-related [16:11:00] dhinus: just for science I'm going to adjust the proxy to bypass the loadbalancer [16:12:41] ...that may have made things even slower [17:11:41] taavi: yes, this was shared with Andrew. These projects have not seen any activity since August and September. [17:26:17] komla: your email was about projects to shut down but aren't there also projects that are due to have their VMs deleted now? [17:26:33] (I thought today's deadline was deletion, not shutdown since shutdown happened long ago) [17:28:03] yeah, it's the same thing. I communicated to the maintainers that the projects will be 'removed'. [17:28:28] I should have made that clear in the email [17:29:14] ok, so really just those two? The other dozen are still getting attention? [17:29:51] Yes, those two for now. Others responded and said they are working on it. I'll follow up again [17:32:57] ok... the spreadsheet needs to be updated then. There are lots of projects on there that don't show 'in progress' and have no notes besides 'Admins notified of Tuesday, September 1st shutdown' [17:35:55] All those of have been shutdown since September 1st and they are not part of the instance count I get from Cumin. [17:36:06] Since the maintainers have not come out to claim them since, I guess they can be deleted. [17:36:18] I was only focusing on the projects that are running now [17:38:05] ok, can you gather up a list of projects to be deleted? The shut-down-but-deleted projects still impose some maintenance burden on us so it's nice to get rid of them. [17:39:20] Alright, cool [17:39:32] thx [18:12:00] andrewbogott: sent. I've also included on the list, the ones you shut down today. [18:12:11] thank you! [18:46:23] * dhinus off