[08:29:37] <blancadesal>	 dcaro: fyi I have deployed ingress-nginx to tools 
[08:29:51] <blancadesal>	 (morning!)
[08:31:18] <dcaro>	 \o/
[08:31:19] <dcaro>	 morning
[08:52:50] <dcaro>	 ci for gitlab seems a bit slow this morning
[13:09:37] <andrewbogott>	 slyngs: thank you for the email w/oidc instructions. Is there a different file in hiera for cloudtestidm or is that configured a different way?  (I'm assuming that cloudtestidm.wikimedia.org is the service that's backed by the codfw1dev ldap)
[13:15:59] <slyngs>	 The IDM isn't what does OIDC, that's IDP (confusing I know)
[13:16:58] <slyngs>	 I don't think we have a cloud IDP, but I'll check
[13:17:13] <andrewbogott>	 thx
[13:18:40] <slyngs>	 There is something called  cloudinfra-idp-1 
[13:19:08] <andrewbogott>	 The context is... our test/dev horizon (and in theory striker) consume a different ldap db: ldap://cloudservices2004-dev.private.codfw.wikimedia.cloud,ldap://cloudservices2005-dev.private.codfw.wikimedia.cloud
[13:19:39] <andrewbogott>	 I think we talked about you building a test sso for that but maybe that got lost in the shuffle.
[13:20:10] <slyngs>	 The  idp.wmcloud.org is just a "proxy" for idp.wikimedia.org, there's some trust between those two
[13:20:38] <andrewbogott>	 ah, ok, so that won't help in this case
[13:21:20] <slyngs>	 Yeah, I have a task for doing the idm for cloud dev ldap, but I haven't seen one for idp, I'll just go hunt in Phabricator
[13:21:21] <andrewbogott>	 what is cloudtestidm.wikimedia.org?
[13:22:01] <slyngs>	 That is half done idm/bitu installation, so that's just managing editing LDAP properties, like the ldapwiki offers
[13:22:24] <slyngs>	 But I need to move that because it's installed in a network that can't reach the dev ldap
[13:22:30] <andrewbogott>	 I think I must not understand the difference between idp and idm. I thought that idm was the UI (e.g. for account creation) and that idp was the thing you actually log in with.  Is that right?
[13:22:38] <slyngs>	 Yes
[13:23:08] <andrewbogott>	 I guess I can imagine how it's possible to have an idm without an idp :)
[13:23:24] <slyngs>	 The other way around works to
[13:24:26] <slyngs>	 We have had IDP for a long time. It just authenticate users via LDAP and provides applications with a OIDC/OAuth2/CAS/REST endpoint, so that users can do single sign on
[13:24:37] <andrewbogott>	 yep, makes sense
[13:25:14] <andrewbogott>	 How big of a project is it to build a new idp with a different ldap backend?  If you put it on cloudweb2002 it should be able to access what it needs to access.
[13:25:16] * andrewbogott double checks that
[13:26:24] <slyngs>	 It is technically just a .deb package, but there is quite a bit of configuration and secrets to change
[13:29:23] * andrewbogott trying to figure out where/how to access that ldap server
[13:29:56] <slyngs>	 Oh, yeah, we need to be able to reach the LDAP server
[13:34:59] <andrewbogott>	 topranks: how hard is it to allow ldap traffic between cloudweb2002-dev.wikimedia.org and the private codfw1dev network?  (e.g. cloudservices2004-dev.private.codfw.wikimedia.cloud)  In addition to needing it for idp testing, I also just now found a different UI that broke when we made that private network :)
[13:35:32] <andrewbogott>	 slyngs: is the idp stuff rolled up into a puppet profile so that it can be added to an existing server with other services? Or does it really need its own host?
[13:35:59] <topranks>	 I think the short answer is that it's simply not possible 
[13:36:19] <topranks>	 it would mean the "private" / isolated network was not, in fact, private or isolated, and just another part of our network 
[13:36:27] <topranks>	 which would beg the question why have it at all?
[13:36:35] <andrewbogott>	 fair :)
[13:36:48] <slyngs>	 The profile can just be added to another host, we've just never ran it an "shared" host
[13:36:55] <slyngs>	 on a
[13:37:18] <topranks>	 the bigger question would then be why the connectivity between them needs to be to the private IP / cloudservices2004-dev.private.codfw.wikimedia.cloud 
[13:37:41] <topranks>	 rather than to 10.192.20.10 / cloudservices2004-dev.codfw.wmnet
[13:37:48] <andrewbogott>	 oh, good point
[13:37:59] <andrewbogott>	 the answer to that is: because I'm copy/pasting without thinking
[13:38:16] <topranks>	 ok yeah, now I know there may be complications there 
[13:38:26] <topranks>	 but I think that's probably the thing we should explore 
[13:38:39] <andrewbogott>	 actually, that seems to route already. So I will see about fixing my config :)
[13:39:00] <topranks>	 what is the connection?  outbound from cloudservices2004-dev to cloudweb2002-dev?  or the other way around?
[13:39:29] <topranks>	 yes routing should be ok.  there are access-lists on the core routers between them filtering ports so we may need to adjust something there 
[13:40:14] <andrewbogott>	 outbound from cloudweb2002-dev
[13:40:59] <andrewbogott>	 https://www.irccloud.com/pastebin/WVuNOATb/
[13:41:06] <andrewbogott>	 seems promising
[13:41:22] <topranks>	 yep that's working 
[13:42:07] <andrewbogott>	 followup question: how hard is it to make a ganeti server that can do that?
[13:42:54] <topranks>	 The networks for VMs are the same as those for physical hosts, so shouldn't really be any different... 
[13:42:58] <andrewbogott>	 ok, great.
[13:43:20] <topranks>	 but I'm not sure I understand the question fully, you mean a new VM?  or you're thinking of a new Ganeti/hypervisor cluster somewhere?
[13:43:31] <andrewbogott>	 Just a new VM.
[13:44:25] <andrewbogott>	 slyngs: so the first thing to try is getting a new idp running on cloudweb2002-dev and consuming ldaps://cloudservices2004-dev.codfw.wmnet 
[13:44:27] <topranks>	 so yeah a VM can be on the public or private networks in production, and if rules are needed they are needed 
[13:44:42] <andrewbogott>	 If that turns out to be a disaster for some reason then we can see about making a fresh VM to host the service.  Sound OK?
[13:45:01] <topranks>	 if it doesn't need internet access then we should avoid the public vlans 
[13:45:32] <topranks>	 andrewbogott: yeah in principal that sounds ok, let me know how you get on 
[13:45:41] <andrewbogott>	 great, thank you!
[13:47:44] <andrewbogott>	 slyngs: you should also tell me if at any point I am not making sense!
[13:48:31] <slyngs>	 No no I think you make sense, I'm trying to plan out the hosts names and "stuff"
[13:50:45] <slyngs>	 andrewbogott: Just so that we're in agreement, it's the server with this role: role(wmcs::openstack::codfw1dev::cloudweb)
[13:51:37] <andrewbogott>	 yep, that's the one!
[13:52:56] <andrewbogott>	 Nothing in codfw1dev is perfectly designed/isolated because we're trying to cram things together on fewer servers. Should be fine to add another thing there as long as there aren't a ton of puppet resource conflicts.
[13:53:06] <andrewbogott>	 And if you need to break that server in the process that is also fine
[13:53:27] <slyngs>	 Cool, there's some minor stuff with the SSL termination, but there has to be a way to make it Apache terminated rather than Envoy
[13:55:05] <andrewbogott>	 I think https://labtesthorizon.wikimedia.org/  on that host is already behind envoy... isn't it?
[13:55:09] <slyngs>	 Oh, nevermind that server already uses envoy
[13:55:15] <slyngs>	 Yes, exactly
[13:55:17] <andrewbogott>	 yep!
[13:55:25] * andrewbogott trying to figure out if the apache there is actually serving anything
[13:55:38] <slyngs>	 The labtechwiki?
[13:55:41] <andrewbogott>	 oh right, labtestwikitech :(
[13:55:51] <andrewbogott>	 which, killing that off is one of the prizes at the end of this race
[13:57:33] <slyngs>	 Hmm,... Why didn't I just attempt to install the IDM on the same host.... ANYWAY job for another day
[13:58:20] <andrewbogott>	 yeah, I guess at this point the idm is lower priority (although we will need a way to create accounts at some point!)
[15:57:44] <dcaro>	 this was unexpected xd
[15:57:49] <dcaro>	 https://usercontent.irccloud-cdn.com/file/cCQXOj45/image.png
[15:58:14] <blancadesal>	 you're a micro-celebrity now
[16:00:14] <dcaro>	 but I was looking for answers and help
[16:00:57] <dcaro>	 now I have to ask myself...
[16:07:14] <dcaro>	 blubber is failing to build 
[16:07:16] <dcaro>	 for me
[16:07:35] <dcaro>	 I think because python now does not allow by default installyng with pip system-wide, looking
[16:08:56] <dcaro>	 hmm... it seems to be blubber-generated pip install command
[16:09:22] <dcaro>	 ` => ERROR [local-python  3/11] RUN python3 "-m" "pip" "install" "-U" "setuptools!=60.9.0" && python3 "-m" "pip" "install" "-U" "wheel" "tox" "pip"`
[16:12:54] <dcaro>	 let me try with the latest blubber/buildkit
[16:13:41] <dcaro>	 hmm... same thing
[16:33:13] <dcaro>	 aaahhh, the latest is v1.0.1, not v0.17.0
[16:33:55] <dcaro>	 this https://docker-registry.wikimedia.org/repos/releng/blubber/buildkit/tags/, not https://docker-registry.wikimedia.org/repos/releng/blubber/tags/
[16:34:49] <dcaro>	 different error now xd
[17:21:39] <dcaro>	 I think I'm going to call it a day, cya!
[20:38:15] <dcaro>	 there were some network issuses that seemed to affect ceph
[20:39:03] <dcaro>	 https://usercontent.irccloud-cdn.com/file/bv68iAMP/image.png
[20:40:16] <dcaro>	 https://usercontent.irccloud-cdn.com/file/CaOlwPHu/image.png
[20:40:27] <dcaro>	 oh, maybe a new node was added and that saturated the switch?
[20:40:50] <dcaro>	 yep !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add :), np, /me back to sleep
[21:40:10] <andrewbogott>	 sorry d.caro -- I changed the cookbook to only do two nodes at once and it still got very busy