[13:24:45] I have been working on this maintenance plan for the dse-k8s-eqiad cluster, to switch out the IPPool. Feel free to give it a read and tell me if you think I have missed anything, or if you just think I'm barking mad. T430658 [13:24:45] T430658: Re-number the dse-k8s-eqiad pod network to 10.68.0.0/17 - https://phabricator.wikimedia.org/T430658 [14:17:18] if anyone wants to pop into the k8s OH to talk about aux, now would be a good time :D [14:26:53] sorry, just saw this, and I have another meeting starting in 3 minutes [14:40:59] np, I'll bother you in a bit :D [14:41:18] and thanks to b.tullis for popping up and telling us all the things about persistent storage :D [14:45:02] I think it was probably a bit https://en.wikipedia.org/wiki/Wikipedia:Too_much_detail [14:45:45] btullis: no more than expected ;-) [14:46:31] it's useful for me to have an overview of all the options, which I didn't have before and do now, so at least I really appreciated it :D [14:49:36] I think we should try and create a support matrix to supplement https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters [14:49:50] excellent idea actually [14:50:37] which leads me to the remaining aux questions [14:51:09] 1. do we want an aux-staging cluster? if yes, do we have any idea when that might happen? [14:52:04] 2. [not for anything concrete rn, just in principle] do we want any kind of "native" persistent storage in aux, or should we tell people to point their stuff at an external cassandra or whatever? [14:55:24] There is already support for the ceph-csi-rbd block devices in aux-k8s. https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/admin_ng/helmfile.yaml#L205 [14:55:43] Not sure if it's used by anything yet, though. I think that it was intended for jaeger. [15:01:52] it's not used by anything yet, it wasn't intended for jaeger but for other potential aux services [15:02:10] one example of something that would be easy to port, we just haven't yet, is the audit logging service for requestctl [16:12:13] cdanis: do you have an opinion on the "should we have staging in aux" question? [16:12:31] (if the answer is "nope", that's fine, I just want to know what to tell people) [16:12:58] not a strong one. I think staging, as we set it up, doesn't serve much purpose. its best function is as an SRE/cluster admin playground. it's really not useful for service owners [16:14:45] hm, interesting, why do you say that? [16:15:13] one of the most important functions of a true preprod/staging environment IMO is access to production or at least production-like backends when you need it [16:15:29] mhm [16:15:34] but the staging cluster as it exists is more like a bunch of stubbed-out RPC servers, with no real dataset behind them either [16:15:38] yeah [16:16:43] most small services that would be good fits for the aux cluster -- like let's consider turnilo or grafana as hypotheticals -- are gonna want their *own* preprod environment on a cluster with some SLA, for e.g. shipping a new UI version and allowing people to actually try it out, with connectivity to some 'real' backend services (druid, prometheus, elastic, wikireplicas, airflow metrics, whatever) [16:17:01] mhm [16:17:26] and you think that's better handled with a separate service deployment in the prod cluster(s) vs a staging? [16:17:44] and I think `foo` and `foo-next` namespaces work really well for that, and should probably exist more on wikikube, but, is already the pattern on the DSE k8s cluster, and seems like a good fit for aux as well [16:17:53] ack, thank you [16:18:04] or at least, is the best model for the kinds of services that are already the best fit for the aux cluster [16:18:47] I think we've also had a service owner or two ask to have a staging/preprod environment with an ingress & CDN plumbing, which is reasonable [16:18:55] and I think we had to turn them down at the time? but I forget [16:48:21] ok, that's useful context, thank you <3 [19:03:28] TIL that dragonfly can just fetch (and in-cluster-p2p) arbitrary URLs of stuff on a lot of different protocols https://d7y.io/docs/reference/commands/client/dfget/