[14:50:20] I'm currently deploying an app called growthbook to dse-k8s-eqiad. Right now, I haven't setup the ingress, as the opensource version does not support OIDC (it's an enterprise feature). Do we have a way to "overlay" an idp connection form on top of an Service, with some Istio magic? [14:53:47] well, not a Service per se, but as part of the ingress resources. Somethin akin to that block [14:53:48] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/f6d10912482f105c492a4e163325f227b021c22c/modules/profile/templates/hadoop/yarn-testcluster.vhost.erb#26 [14:55:01] brouberol: I don't think we have anything on the ingress side unfortunately (perhaps we should) [14:55:11] we run some oauth2-proxy sidecars elsewhere though [14:56:10] heh but on k8s, only on aux, from the upstream jaeger chart [14:59:12] yep, a cursory search led to https://github.com/mosn/envoy-go-ldap-auth [14:59:25] which is a self-proclaimed "do not use in production" envoy plugin [15:00:21] I think this approach uses Istio features we don't ship on wikikube but maybe that's not true on dse? https://developer.cyberark.com/blog/istio-oidc-authentication/ [15:03:33] IIRC we have a very vanilla set of istio features in dse [15:03:35] yeah, the mesh extensions can't be used on wikikube since there is no mesh (only the ingress). But the oauth2-proxy should generally be a good option I'd say [15:05:06] jayme: do we happen to already ship all the required istio CRDs? [15:05:40] ah, sorry, I just connected what you said to what cdanis said. cf jaeger [15:05:42] brouberol: yeah, those should be complete regardless of the actually enabled istio features [15:05:57] ok, so we have a lead then! Thanks y'all, as always [15:06:35] I don't think you need the CRDs for a roll-your-own approach? [15:06:45] but I guess you'd still have to (re)configure istio on the cluster for the extension [15:09:03] > I don't think you need the CRDs [15:09:03] Yep, I wasn't clear that this was a custom option. I've checked out the jaeger chart, and it's clearer now [15:10:08] IMO we should be looking at moving to Gateway API instead of ingress [15:10:37] jayme: could you expand a bit? I'm not super familiar with istio, and how it's configured here. The whole CNI side of Kubernetes is not something I've been much exposed to, except for the network polices and ingress gateways [15:10:55] https://gateway.envoyproxy.io/docs/tasks/security/oidc/ [15:11:07] doesn't help now of course :) [15:11:41] brouberol: so, the post I linked uses Istio VirtualService https://istio.io/latest/docs/reference/config/networking/virtual-service/ amongst other things [15:12:05] cdanis: yeah, we went with istio crd since gateway api wasn't really up to the task back then [15:12:06] I'm not sure how much closer that is to the 'ingress' side of things than the 'actual service mesh instead of identically-configured Envoys' [15:12:11] jayme: yeah, ofc [15:12:24] I use traefik CRD at home still 😅 [15:13:49] (I'm reading https://medium.com/@lucario/istio-external-oidc-authentication-with-oauth2-proxy-5de7cd00ef04#f525, which _seems_ to address the istio reconfiguration aspect) [15:14:00] (gawd what a horrid website) [15:15:26] brouberol: right. the link c.danis send earlier does as well, but via the IstioOperator (which we don't use). But in generall it's possible to translate it's configuration to other installation methods (like the "config profiles" we're using) [15:15:48] deployment-charts/custom_deploy.d/istio/dse-k8s/main.yaml - in case that reference is missing [15:18:24] dse-k8s/config.yaml *? [15:28:57] oopsie [15:29:00] yes, sorry [15:29:23] that's the istio config that is deployed (as you already figured) [15:36:06] gotcha thanks [15:40:03] cdanis: just looping back on jaeger 2s. I see that the oauth-proxy is running as a sidecar pod the jaeger pod. How are we integrating that upstream chart to our ingress gateway? [15:40:30] Oh, maybe we're _not_ integrating it, and jaeger gets its own ingress VIP, mapped to a nodeport service? [15:41:21] brouberol: no, we are -- check out helmfile.d/aux-k8s-services/jaeger/values.yaml particularly the `extraObjects` key [15:42:06] ooh, you bundle your own representation of the istio ingress resources [15:42:13] how delightfully yaml [15:42:22] yes, and with template expansion even [15:42:59] one day I'll do a lightning talk about https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/airflow/templates/_helpers.tpl#216 [15:43:14] ahaha [15:43:15] and will quit right after as I'll automatically become un-employable [15:43:40] https://i.imgur.com/098Ll3J.png [15:44:22] alright, so that means that we're doing ingress-gateway --> oauth2-proxy sidecar --> jaeger [15:44:32] did I get that correctly? [15:44:55] correct [15:45:05] and oauth2-proxy is just running in reverse proxy mode pointing at localhost [15:45:30] and it's provisioned with a secret etc as its own OIDC client [15:45:52] right, gotcha. hence the `upstreams = ["http://localhost:16686"]` config [15:47:10] aye [15:51:08] might now be the time to backport the ingress stuff from jaeger to the ingress chart module? 🤔 [15:53:00] jayme: what's the best way to do that without hacking up the upstream chart too much? [15:53:42] that's the question... and probably part of the reason we did not do it in first place? I don't recall exactly [15:53:53] yeah I think so [15:54:05] perhaps we could add on a library chart of our own [15:55:28] is there a way to run this a a service, global to the whole k8s cluster? [15:55:34] aka it being a chart of its own [15:55:56] that would be a completely different path to the chart modules, not sure if it's a good idea to have yet another method of doing things [15:56:43] brouberol: not easily [15:56:43] fair [15:56:52] IMO that's what migrating to something like Gateway API would buy us [15:57:17] but I'm not mistaken the ingress config from jaeger should not be relevant for the use of the oauth2-proxy...or am I missing something? [15:57:53] ok, so it must run as a sidecar container, meaning that we could try to facilitate adoption via some chart modules, depending on how hard it is [15:57:56] yeah, it's not -- that's just the ingress config you need to bring for any service [15:58:05] (and jaeger is like three or four services heh) [15:58:35] brouberol: a chart module that makes an oauth2-proxy sidecar easy would be good I think [15:59:13] +1