[12:09:19] slyngs: I am briefly awake! (but have an appointment in an hour, as I think you do). How are things going? [12:17:41] "Small" issue, the cloud web server is Buster I think and the IDP needs Java 21 [12:18:06] Otherwise I think we're almost there [12:19:17] There are two options, either I revert all the removal of support for CAS 6.6 (we're running 7.0) and rollout an older version or we upgrade the server. Not sure which is the easiest [12:19:49] Dang, it's really still buster? I definitely already upgraded the other cloudwebs... [12:20:22] Sorry, bullseye... I'll just check [12:20:39] Yeah, bullseye [12:20:50] and I don't think we can run labtestwikitech on bookworm [12:21:26] Then we can jump Moritz when he's back Monday and ask if we can build Java 21 and Tomcat 10 for Bullseye [12:22:07] CAS 6.6 isn't terribly old and the parts you need are the same, so that might be easier [12:22:40] If it's just a patch revert then that seems ok... [12:22:59] we can also change course and put this on its own ganeti instance [12:23:21] We'd need to revert https://gerrit.wikimedia.org/r/c/operations/puppet/+/1066708 [12:24:06] Hmm, maybe just parts of it [12:24:51] You know, let me just test somethinh [12:24:56] ok :) [12:37:27] That worked... There probably something missing: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1068786 [12:38:03] So this would need to be merged first: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1069165 [12:41:38] So it's the same role, just with overrides based on the host... [12:42:14] Yes, there is some overwrites defined for the cloudweb2002-dev host [12:42:16] the secret for keystone/horizon auth will need to be shared across a few hosts but I guess we can separate that by datacenter or something [12:43:08] but otherwise this will use all the same secrets and things in private idp.yaml, is that risky? [12:43:51] We overwrite the service and secret for each "installation", so idp, idp_test and cloudweb2002-dev share nothing [12:44:15] So in this file: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1068786/8/hieradata/hosts/cloudweb2002-dev.yaml [12:44:51] We define a "Horizon" OIDC service, that exists only on cloudweb2002-dev and the secret is going to be unique to that installation [12:45:20] I'm thinking about all the things in that private file like 'profile::idp::webflow_encryption_key:' and 'profile::idp::web_authn_encryption_key' etc etc [12:45:45] They are overwritten for the cloudweb2002-dev hosts as well in the private repo [12:45:56] oh, great! ok [12:46:13] I did that for the dummy secrets already: https://gerrit.wikimedia.org/r/c/labs/private/+/1069114 [12:46:50] great! [12:47:30] so let's see what the pcc thinks [12:48:17] Are we brave enough to merge this https://gerrit.wikimedia.org/r/c/operations/puppet/+/1069165 on what is a Friday afternoon for me? [12:49:08] I'm not sure :) Only you know for sure how risky it is... if it failed I'd expect it to fail immediately [12:50:18] But you're about to step away, right? So we should either do it in an hour when we're both back, or Monday [12:50:31] I'd enjoy having this to tinker with over the weekend but it's not required. [12:51:03] I'd kinda prefer doing it on Monday. I am back in about an hour, but I'm also alone with a 5 year old [12:51:34] I'll just make a quick change, maybe that will make it same not risky [12:51:38] ok, fair [12:53:37] Okay, so this makes it do nothing in production: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1069165 [12:55:21] Aagh, I still need to figure out the Envoy stuff. Cloudweb2002-dev already had Envoy so the default IDP config doesn't work [12:56:19] I'm sorry, are you okay with Monday... I know now would be better [12:57:00] Yeah, Monday is ok! [12:57:57] I need to go in a minute anyway. Will keep things in Gerrit for now. [12:59:49] I dont think anything would break, but I'm also not sure that Envoy would be configured correctly, so it wouldn't work anyway [13:00:05] I have to run, I'll just check back with you when I'm back [13:59:36] * andrewbogott is back [14:01:38] Also back. I merged one of the patches, seen as it does nothing [14:01:43] In production at least [14:01:56] I've also added the secrets for cloudweb2002-dev [14:02:38] I'm fairly convinced that it would work completely, due to Envoy and certificates missing. Is that something you can fix yourself, or trick someone into giving a go? [14:02:51] would not [14:03:00] Less optimistic :-) [14:07:25] I also don't really want to mess with Envoy on a Friday so I think this will need to wait. I will see if I can get Horizon as far as trying to redirect to the unreachable service though :) [14:09:51] Cool, I'll finish up on Monday then. What you can do it take say idm_dev on idp-test.wikimedia.org or some other OIDC service on test and provide those credentials. CAS WILL reject you and be mad about, but that should allow you to make a qualified guess as to whether or not everything is working [14:10:55] oh yeah, good idea [14:11:14] Then you can do journalctl -f -u tomcat10 on idp-test1004, it has debugging enabled and that will allow you to make a pretty good guess about what is working and what's not [14:12:05] Oh, except what about the client_secret? When does idp check that? [14:13:53] I can't remember if it check your callback first... It might check the callback first, and then the secret [14:15:02] ok, I guess it'll be obvious when I get there [14:15:42] You can try borrowing the secret from idm_dev (it's on idm-test1001.wikimedia.org, in /etc/bitu/settings.py under SOCIAL_AUTH_CAS_SECRET) I can just reset it on Monday if you lose it :-) [14:17:27] ok! [14:18:20] thank you for throwing this together on short notice. Hopefully we can snap the pieces together next week. Monday is a US holiday but I'll at least poke my head in. [14:19:50] Cool, I have a "long" day on Monday, so there's more overlap time-wise [14:20:10] Have a great weekend when you get there :-) [14:20:46] thx