[09:43:43] dcaro: this is a new attempt to what I tried yesterday (but had to revert) https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/37 [09:59:24] thanks [10:19:06] np [10:19:08] * dcaro lunch [11:16:31] * arturo back in a bit [12:40:53] quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/315 [12:41:51] +1'd [14:07:40] andrewbogott: for T365696 I need an account with domain-wide admin rights. I probably should register a new account instead of re-using novaadmin. should I do that just as a normal ldap/developer account or is there a way to register some keystone-only service account? [14:07:41] T365696: Investigate how to run OpenTofu to manage Cloud VPS admin-only resources - https://phabricator.wikimedia.org/T365696 [14:12:09] taavi: if it's for the default domain then it needs to be in ldap [14:16:19] ok! .. how does one create a new account in the codfw1dev ldap these days? [14:17:05] :-( --> FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 28.63M minutes old [14:17:17] ooh the answer to this is embarassing. I don't think bitu is setup yet (slyngs might know otherwise) [14:17:38] so either you can add it to ldap directly, or you can use labtestwikitech. BUT to use labtestwikitech you have to enable account creation in the config first [14:17:43] which I will do right now :) [14:18:25] andrewbogott: I'm not sure we can do that anymore, or whether we've (= I've) already ripped too much things out of the config to make that work [14:18:33] arturo: hmm, sounds like the `for:` time is still too low? [14:18:34] oh, good point [14:19:05] taavi: yeah, something weird. I'll keep researching [14:19:36] taavi: so I guess edit ldap. I was doing that all day yesterday so I have the commands in my history, what username/shellname would you like? [14:20:09] cloudvps-admin-tofu or something similar? [14:22:33] I'm superstitious about dashes, mind if it's just tofuadmin? [14:23:20] what's wrong with dashes? :-) [14:23:29] probably nothing! [14:23:35] taavi: turns out, https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/14 was never merged :-P [14:24:12] (just merged now -- the `for:` change) [14:24:19] that would do it [14:24:20] https://www.irccloud.com/pastebin/tk0gXHqC/ [14:24:39] taavi, I'm giving it your personal email so it's easy for you to do a password reset (assuming that that bit is still possible) [14:24:58] not without a bitu instance :-) [14:25:38] well, ok, you'll need to set the password using ldap cli then [14:25:45] otherwise it's done [14:25:49] ok, thanks! [14:26:03] it might work! lmk if it does [14:26:14] I'll have a look after our meeting [14:44:24] andrewbogott: Yeah, I created an LDIF for myself as well [14:44:59] ok :) We'll all look forward to having a gui [14:45:51] andrewbogott: Regarding Bitu, there's some network related issues that prevents the VM for accessing the LDAP server for DEV. So I was considering moving Bitu to cloudweb2002-dev instead. How would you feel about that? [14:46:06] I keeps the network nice and clean, but adds an additional service to that host [14:47:38] Oh, good point re: r/w ldap [14:47:57] yeah, moving it there is probably the simplest path [14:48:59] it runs in a docker container? [14:49:04] It can [14:49:28] We do have one, so if that's preferred no objection on my part [14:49:40] actually, wait, when you say 'the VM' are you talking about a ganetti VM? [14:49:47] Yes [14:50:31] huh, I'd expect networking to work OK then, can you tell me what the problem is? [14:51:04] XioNoX can probably do it better, but there's some additional firewalling keeping those bits of the network separated [14:51:44] ah, so it's just not possible to have ganetti hosts talk to the cloud-vps private network? dang [14:52:09] It's possible I think, but is it nice ... I don't know [14:52:39] I just requires exact steps, which I think we have done for some services [14:52:42] It [14:53:12] both ganeti vms and cloudwebs are outside the cloud-hosts/cloud-private networks, so they would have the same limitations [14:53:54] Also the -dev webses ? [14:54:06] yes, both are hardware in the public vlan [14:54:27] cloudweb-devs probably already have a hole to talk to the codfw1dev ldap, poking one for the bitu vm should be a matter of adding a few lines to the homer repo [14:55:31] thanks taavi, that's what I was about to say: it's the same problem for cloudweb as for ganetti as far as I know [14:55:44] um... ganeti [14:56:22] taavi: do you have access/knowledge to edit homer or should we bother cathal? [14:57:43] I can go poke either Cathal or Arzel [14:58:03] i can also do it, although getting a +1 from the would not be a bad idea [14:59:02] No I think we should, just so they know [14:59:52] topranks: we're talking about adding an exception so that a new ganeti VM (codfw1dev bitu) can talk to ldap on the cloudservices2xxx hosts, do you have any concerns? [15:00:21] (this is just piling test/dev exceptions one on top of another) [15:01:02] andrewbogott: I'd probably need to think it through, does the VM exist already? [15:01:13] or more to the point what network/vlan would it be on? [15:01:42] topranks: Yes, the VM exists [15:01:56] what's the name of it? [15:02:23] What we'd like to have working is [15:02:24] nc -vzw 2 cloudservices2004-dev.codfw.wmnet 389 [15:02:33] cloudidm2001-dev.codfw.wmnet [15:03:36] basically that would be https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1039742 [15:04:14] topranks: the premise here is that we want a toy bitu that can read/write to our toy ldap, in order to replace a bunch of labtestwikitech functionality. [15:04:20] (that was maybe obvious) [15:06:00] I'm not overly familiar with what a "bitu" is tbh :) [15:06:22] what is "cloudidm" ? [15:06:49] Installation of the idm.wikimedia.org software for use in the Striker/labs environment [15:07:11] but it's going into the WMF production ganeti rather than into the labs environment? [15:07:12] * arturo offline [15:08:00] I did the installation on a production Ganeti, but it needs to manage the labs LDAP. [15:08:35] There's also the option of redoing the installation somewhere in the labs environment, if we don't like the network changes [15:10:49] when we need to make exceptions that's ok and we can do it [15:11:01] but overall that sounds like the wrong sort of shape [15:11:16] if we need to create a new cloud service to manage cloud elements it should go into the cloud realm [15:11:29] rather than being put in wmf production and then creating exceptions to allow them to talk [15:12:01] codfw1dev is test/dev but it runs on hardware [15:12:45] IMO having something on the cloud have r/w ldap access breaks more rules than having bitu on ganeti :) [15:14:19] Ideally I'd like to have it within the same network: 10.192.20.10/24 [15:14:28] With all the other dev things [15:14:51] if we could do a (ganeti) vm in cloud-hosts/cloud-private that would be the best option imo, but i don't think we can atm? [15:16:16] to be clear, I'm not strictly opposed to it moving to labweb2xxx which already has the network exception. Just seems like more trouble/less service separation [15:17:58] Did I get the hosts wrong, I thought that the labtestwikitech installation was on cloudservices2004-dev [15:18:35] the mediawiki installation is on cloudweb2002-dev [15:18:59] ldap is on cloudservices200x-dev [15:19:01] cloudservices2004/5-dev house the LDAP database that both labtestwikitech and the new bitu installation will talk to [15:19:15] Sorry, yes, cloudweb2002-dev. [15:20:07] it seems like this new function ought to run somewhere on the cloud networks [15:20:28] but cloud services have no infra to run VMs? and don't want to add new physical hosts for it? [15:20:45] Seems like a lot to waste a physical host on [15:20:55] the other way.. seems like to little [15:21:16] But we can just run in on cloudweb2002-dev. andrewbogott if you're okay with it [15:21:17] I'd rather solve the problem of "how do we run this in the cloud network" than to create it now on prod. infra and have another exception and element we probably can never separate [15:21:57] cloudweb2002-dev is also not in the cloud network fwiw [15:23:00] already have the right networking setup and firewall rules [15:23:23] but we need to untangle cloudweb's this is just making our life worse [15:23:37] We don't want that :-) [15:23:50] sorry I don't want to make anyone's life difficult [15:24:18] No no, it fine, we would have asked if we didn't want your opinion [15:24:26] wouldn't [15:24:53] topranks: it's a bit of a circular dependency :-) this unblocks part of not making wikitech be a ldap wiki, which in turn unblocks moving cloudwebs to the cloud-hosts/private setup that other cloud* hardware uses [15:26:00] taavi: ok, well if it gets us towards that then it's definitely good [15:26:15] if that's the case it's probably fine on cloudweb2002-dev if it can run there? [15:26:30] and then when we get to the end of that untangling it would move into the cloud rack along with cloudweb? [15:26:31] Maybe the easiest then would be to run the code as a container on cloudweb2002-dev, then we can just move it, when/if that because an option? [15:27:13] i'd be fine with that [15:27:22] andrewbogott: ? [15:28:19] yeah, if it's containerized it shouldn't make much more of a mess than we have already [15:28:45] I'll make it work in a container [15:28:54] striker and horizon are already running there in blubber-built containers, the puppet code for deploying that should be obvious [15:29:39] taavi already made me build blubber-built containers for Bitu, so we have that ready. Might need a tweak or two, but that's easy enough. [15:30:06] cool [15:30:20] I really need to run, but thank you everyone. I'll make the needed changes to Bitu/container images and Puppet [15:38:16] andrewbogott: do you know if the user id that keystone uses in it's authentication is the 'username' that you enter in the login page of horizon? https://docs.openstack.org/api-ref/identity/v3/index.html#password-authentication-with-scoped-authorization [15:38:55] I think it can auth by name or by id [15:39:06] But I believe the id is the shell name [15:39:27] e.g.: [15:39:31] https://www.irccloud.com/pastebin/KhoTuYQB/ [15:39:36] dcaro: is that what you were asking? [15:39:43] yes :), that's useful [15:40:31] for application tokens though, you have to pass both, the secret (the token) and the application id? or do you know if you can just pass the token itself? (feel weird that you need both, not a common 'token' auth) [15:42:07] for an application credential, you need both the cred id and secret, yes [15:43:43] I'm thinking on using a bearer token auth header, and embedding both like `:` so clients can use `Authorization: Bearer :` header (with the token being the chained string) [15:44:17] that way most libraries will not need anything weird, does that make sense? 🤔 [16:06:35] taavi: don't app credentials get you a token which you can use on its own for subsequent calls? (for, like, X number of hours) [16:06:49] yes [16:07:45] maybe that's what david was saying, I'm distracted by staff meeting [16:18:37] not really, you still need the app credentials first [16:21:42] true [16:23:34] Toolforge clients don't need to deal with keystone, they deal with toolforge, then toolforge deals with keystone xd [16:23:38] (or whatever) [16:30:32] *should not need to [17:02:06] * dcaro off [17:02:11] cya on monday