[06:39:52] greetings [06:48:24] I have started an announcement draft at https://etherpad.wikimedia.org/p/T404584-announcement re: tools nfs upgrade, please edit at will and/or let me know what you think [07:15:42] godog: I added a sentence about the expected impact [07:16:08] taavi: great point, thank you [07:28:46] mmhh are cinder volumes expected to work in codfw? https://phabricator.wikimedia.org/T406688#11253424 [07:33:20] yes [07:34:50] ok thank you, I think I can see the rabbit hole from here and I'll wait for Andrew if he has some immediate lead or insight [08:00:32] morning [08:01:21] o/ [08:02:02] hello [08:08:29] just added a note about tools-static [08:10:47] sweet thank you! [08:46:53] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/226 [09:19:05] LGTM [09:19:19] FYI I'll be shutting down nfs on toolsbeta while investigating T406688 [09:19:20] T406688: Cinder volumes getting stuck on 'reserved' after detach - https://phabricator.wikimedia.org/T406688 [10:36:16] ack [11:44:33] * dcaro lunch [13:04:20] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/278 [13:42:28] Raymond_Ndibe: for the reviews of jobs-api/component-api, I've linked them all from the parent one here https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/140 [14:07:40] fun fact, we have three unused servers (cloudcontrol100[8-10]-dev) in eqiad connected to cloudswitches waiting for a purpose. So we could quite easily use them for a baby k8s cluster. [14:07:51] it would probably be best to rename them firrst [14:10:27] akosiaris: ^^^ andrewbogott what are the specs? compatible with it? [14:23:59] volans: sorry, was on a call. They're config B: 10 cores, 64GB of RAM. So we wouldn't want to use them for any full-blown workloads but they would let us get a proof-of-concept going without waiting for new hardware delivery. [14:24:20] And help us iron out networking questions which (in my head at least) is the biggest technical challenge [14:24:47] ack, yeah 64GB is not much [14:57:06] dcaro, before you go can you comment on the object-storage requests on https://phabricator.wikimedia.org/project/board/2875/ ? The resources needed are small but I'm unclear on if creating a separate tenant for tool object storage is something new or something we've already done for other tools... [14:57:18] obviously it's not the best long-term approach but maybe it's the best we've got today [14:57:34] 👀 [14:59:13] I think we have a cookbook to create a db-only tenant but I don't know that we have one for a storage-only tenant. I'll make one if this is something we're going to be doing. [15:01:36] we have some tenants per-tool, but I'm not sure if they are more like several related tools use them [15:02:12] if it's not too much trouble, I'd create both, it's a bit annoying for the user, but they are requesting it [15:02:41] yeah, I'm convinced that it should be per-tool, I just wasn't sure if you had a better storage solution days away from release :) [15:03:07] go ahead and +1 and I'll create [15:03:08] ah, no, the storage side of toolforge is hanging pending resources xd [15:04:04] (as in someone tackling it) [15:05:21] 'k [15:06:33] done [15:08:13] ty [15:41:18] I'm off a bit early, I'll be back later [15:41:25] but otherwise cya on monday! [15:49:49] volans: just saw this. They are probably overpowered for a control plane, so perfectly fine for what we want to do. [15:49:52] andrewbogott: ^ [15:50:48] yeah, def big enough for a proof of concept [15:51:30] If anyone wants to suggest new names for them I can work on getting them renamed and puppetized. (Also, dcops wants to relocate one of them so probably best to get that out of the way.)