[09:43:41] morning [09:44:43] argh, pydantic requires python >=3.8, the CLIs use 3.7. (wanted to use it for validation in the components-cli) [09:48:57] xd, working on deprecating the last bastion... [09:49:09] *last buster bastion [09:50:39] in the meantime... should we 1) skip validation (we only want to use this for internal testing for now) 2) use something else like jsonschema 3) ? [09:55:16] I'd skip validation for now [09:55:36] 👍 [09:55:37] it's the simplest common denominator [09:55:52] (as in, we will not have to undo anything) [09:57:10] yep [10:13:00] I need a +1 for this quota request: T379556 [10:13:00] T379556: Increase RAM and storage for webperformancetest - https://phabricator.wikimedia.org/T379556 [10:16:05] +1d [10:16:30] thanks [10:17:11] dcaro: what's the strategy for the first debian changelog? [10:17:19] https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/jobs/396064 [10:17:47] blancadesal: hmm... I think you might have to create it manually [10:28:15] that worked! [10:42:11] \e/ [10:43:59] somebody has read my mind and written a feature request I've been thinking about many times :) T379704 [10:43:59] T379704: Function infrastructure for Cloud/Toolforge ("Wikimedia Cloud Functions") - https://phabricator.wikimedia.org/T379704 [10:48:47] dcaro: skeleton-review 👻 https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/1 [10:50:07] dhinus: sounds like another topic for the offsite :)) [10:58:26] blancadesal: indeed! [10:58:44] definitiely! [10:59:07] *definitely (or whatever...) [11:23:32] dhinus: I just saw T379727 but I don't think that is the right way to go [11:23:32] T379727: Replace legacy domain docker-registry.tools.wmflabs.org with docker-registry.toolforge.org - https://phabricator.wikimedia.org/T379727 [11:23:58] the FQDN `docker-registry.toolforge.org` already exists, as a standard toolforge webservice tool [11:24:16] the FQDN for the registry endpoint itself should contain the `svc` in my opinion [11:24:16] hmmm I saw the wmflabs now redirect to that one [11:24:32] so I assumed that was the current "official" one, but maybe it isn't> [11:25:00] I will update the task to be more generic and just say "remove legacy domain" [11:25:11] then we can discuss in the task what we should replace it with [11:26:37] here it was decided to use .svc. T306039 [11:26:38] T306039: Decision request - Toolforge external infrastructure domain usage - https://phabricator.wikimedia.org/T306039 [11:27:11] (I'm open to discussing any exceptions, ex. I think that user-facing services might benefit from having a more memorable name) [11:27:38] I think T366453 is still valid and should be reopened, and T379727 on the other hand resolved as invalid [11:27:39] T366453: toolforge: introduce docker-registry.svc.toolforge.org FQDN to replace docker-registry.tools.wmflabs.org - https://phabricator.wikimedia.org/T366453 [11:28:11] arturo: ok, sounds good to me [11:28:24] +1 docker-registry is not meant to be used directly by users [11:28:44] so [11:28:55] so docker-registry.toolforge.org should also be removed? [11:29:21] `docker-registry.toolforge.org` is just a toolforge webservice tool, an UI to show the registry content [11:29:29] it should remain at it is [11:29:40] similar to `openstack-browser` and others [11:30:30] ah ok so I completely misunderstood what docker-registry.toolforge.org is, just a frontend for public consumption [11:30:35] i.e, you can `become docker-registry` [11:31:45] I agree the naming can be confusing. And maybe that's another reason that supports having the `svc` keyword in the FQDN [11:32:08] (for the registry endpoint itself, I mean) [11:32:54] I think it's ok, I was just confused by the redirect that was added in T375515 [11:32:55] T375515: docker-registry.tools.wmflabs.org returns the nginx default page - https://phabricator.wikimedia.org/T375515 [11:35:38] apologies for the confusion, I closed T379727 as invalid and reopened T366453 with an additional comment [11:35:39] T379727: Replace legacy domain docker-registry.tools.wmflabs.org with .toolforge.org - https://phabricator.wikimedia.org/T379727 [11:35:39] T366453: toolforge: introduce docker-registry.svc.toolforge.org FQDN to replace docker-registry.tools.wmflabs.org - https://phabricator.wikimedia.org/T366453 [11:36:34] does that apply to harbor? [11:39:02] does harbor have other domains apart from tools-harbor.wmcloud.org? [11:39:14] nope [11:49:04] hmm reading T306039 maybe we should migrate tools-harbor.wmcloud.org to harbor.svc.toolforge.org, but it's a bit different than the other ones (prometheus, nfs, etc.) because it also has a web interface [11:49:05] T306039: Decision request - Toolforge external infrastructure domain usage - https://phabricator.wikimedia.org/T306039 [11:50:45] yeah I would go with `harbor.svc.toolforge.org` [11:51:04] I just noticed that https://prometheus.svc.toolforge.org/ is returning an apache default page, and https://deb.svc.toolforge.org/ is returning an nginx default page :) [11:51:05] the registry web interface is just a toolforge tool [11:51:35] should harbor.svc.toolforge.org return the harbor web interface? [11:51:42] yes, why not? [11:51:57] I mean, it is part of harbor anyway [11:52:17] the docker registry doesn't have a web interface I guess, that's why a toolforge tool is in place [11:52:18] yes, it's just "different" from the other "svc" services, but it's not a big problem [11:54:46] I think svc just indicates that the FQDN is meant to be 'internal' in the sense of being part of the infrastructure. It doesn't imply anything about if the FQDN is meant to be consumed via a web interface, an API or whatever [11:55:21] I think prometheus.svc, nfs.svc, deb.svc should probably not listen at all on 80/443 [11:56:58] * dhinus lunch [11:56:59] yeah [12:19:03] quick rev': https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/593 [12:37:53] another: https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1090839 [12:47:10] both +1'd [12:54:23] thanks! [13:06:16] tofu review https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120 [13:08:51] +1'd [13:09:40] cheers [13:18:35] thin one follows from that one now https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/112 [13:33:29] and testing the full cookbook https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/123 [13:51:27] fix for missing packages: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/594 [13:52:24] 👍 [14:03:37] dcaro meeting ping [15:59:15] dhinus: can you forward the AWS abuse email to me? I don't find it in my inbox [15:59:42] arturo: sure [15:59:56] thanks! [16:00:02] it's actually entirely pasted in the task T379759 [16:01:03] oh, ok! [16:01:19] I misread the email, as if there were additional details attached or something [16:25:56] andrewbogott: I have this patch: https://gerrit.wikimedia.org/r/1090854 wdyt? [16:35:18] arturo: I can't decide. I sort of like the simplicity/uniqueness of the uuids but it sure is unfriendly to the eyes. [16:35:38] What will happen to existing uuid projects if we make that change? [16:36:03] I think we would want to backfill the "broken" group names [16:36:09] we will need to create a script to somehow backfill them in LDAP [16:36:12] I'm updating the gitlab token for cookbooks to one that can create MRs, let me know if you find issues [16:36:22] (cloudcumin only) [16:36:24] Or I rewrite stashbot and a several other things I guess [16:38:42] oh, this is new, cookbooks don't find gitlab.client lib [16:38:49] dhinus: ^ had you seen that before? [16:38:55] not that I remember [16:39:26] we were using that lib before, but I deployed stuff not that long ago, not sure when it got broken [16:39:33] arturo, bd808, I'm watching the meeting right now, will need to think about this when I can pay attention. I'm confused about which things/why things are consuming ldap rather than keystone when enumerating projects. [16:40:07] https://www.irccloud.com/pastebin/FPOo6Vqo/ [16:40:16] Interesting, I'll open a task and fix it [16:40:25] andrewbogott: sure, no rush [16:42:24] T379782 [16:42:24] T379782: [cloudcumin] cookbooks fail to load gitlab.client - https://phabricator.wikimedia.org/T379782 [16:42:30] birgit just said a sentence including the phrases 'beta cluster' and 'taking on ownership for' [16:44:06] andrewbogott: Stashbot reads projects and tools from LDAP for !log processing. Humans do look at this data too. Sprinkling the UUID into various places like LDAP and DNS feels like an accident rather than a deliberate choice, but I'm ready to hear how that is a bad take. [16:46:10] in the meeting [16:46:47] stef just said 'patchdemo.wmcloud.org', imagine instead having to say '13298h18y430123.wmcloud.org' heh [16:51:01] andrewbogott: you may be about to find out that WMCS is part of Developer Experience ;) [16:51:32] quick review https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1090904 [16:51:33] I'm aware! But I'm hoping there are tactics as well as strategy [16:51:48] or better, that WMCS is now the de-factor owner of deployment-prep heh [16:52:22] arturo: in theory dns entries are created using name + id both [16:56:10] dcaro: +1d [16:56:21] thanks, waiting for the tests... [16:59:01] * arturo offline [17:15:32] huh, cloud-announce just bounced my email [17:28:07] andrewbogott: let me know if you need me to check config or something [17:28:32] I rearranged it and re-sent and it seemed to work, did you get it? [17:29:07] I also removed the phrase 'reign of destruction' which maybe is a spam trigger for some reason :p [17:30:51] xd, yep got it [17:43:59] andrewbogott: I seem to be unable to login to labtesthorizon using idp, do you remember if there's anything I have to setup anywhere? [17:44:10] (afaik that has not changed since you left xd) [17:48:55] I just got logged in there dcaro, but it did seem to redirect a few more times than normal (or the redirects were slow) [17:49:42] maybe I'm using the wrong user? does the same as prod idp/ldap work? or should I create another one somewhere else? [17:49:46] I have definitely not tried it for months, let me try now... [17:50:33] My user there is `BryanDavis-labtest`, but I don't remember if that is in a different LDAP tree or not. [17:50:58] yeah, different ldap [17:51:11] dcaro: is it just rejecting your pwd? [17:51:44] Yep, I also tried the user/pass I used when it was not using idp, let me retry just in case [17:52:13] It should be the same user/pass that it always was for labtest [17:52:23] just checked via a different backend [17:52:26] oh, `I got You've hit an OpenID Connect Redirect URI with no parameters, this is an invalid request; you should not open this URL in your browser directly, or have the server administrator use a different OIDCRedirectURI setting.` [17:53:39] ...no idea what that's about. Maybe your browser treating the redirect differently? [17:53:44] Has it ever worked for you since moving to idp? [17:53:46] going back to labtesthorizon now seems to get in a redirect loop [17:53:52] I had never been able to login no [17:55:03] It definitely redirects too many times but then works for me [17:55:06] I think that https://phabricator.wikimedia.org/P71035 is the "best case" set of requests to get logged in [17:56:02] it's bouncing through those urls yep, changing the token/nonce/... every time though [17:56:07] still looping [17:56:22] so far >500 requests in total [17:57:22] oh, dropped me to the cloudidp-dev login page [17:57:23] xd [17:58:31] gtg, will try later, it's looping redirects again after entering the user/pass, we'll see when it finishes [20:14:51] There use to be a stewards channel here, is there still one? [20:20:21] Oh there it is, please disregard [23:38:00] Rook: without checking the technical details yet, of the accounts that you just posted Skywiks stands out as different - they have what appears to be useful contributions on frwiki. are you sure they have mining software on PAWS? [23:39:16] Let me check. Might have missed something [23:42:58] Yep, they just got caught up in the mess of other ones that were registered in the last few days. Their home directory only had some untitled things so I missed it among the ones