[08:14:23] Emperor, godog, all codfw B4 services are depooled, going to reboot the switch for upgrade [08:14:50] ack [08:14:54] XioNoX: ack [08:29:36] all done! [08:29:48] repooling services (cc moritzm ) [08:30:48] ack [08:34:25] neat [08:34:57] ms swift in codfw looks good. [10:36:20] hey oncallers [10:36:34] I'd need to restart the druid indexing job for webrequest_sampled [10:36:48] https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Druid#Sending_the_indexation_task [10:37:04] the new job will add a couple of extra fields [10:37:08] ok to proceed? [10:37:26] (I mean, I don't see anything ongoing but lemme know otherwise, this could affect turnilo/superset) [10:41:27] yes go ahead elukey [10:41:33] thank you for the heads up [16:11:20] Does anyone know how set up discovery to pool/depool an individual service that's behind k8s ingress at DC level? Like we have `k8s-ingress-dse-aa.discovery.wmnet` but we'd like to be able to be more fine-grained? [16:11:35] cc btullis brouberol [16:14:12] IIRC this is using anycast to resolve to the local ingress VIP, so AFAICT the scope is the ingress IP, for all services [16:17:51] it's defined with `k8s-ingress-dse-aa 300/10 IN DYNA geoip!disc-k8s-ingress-dse-aa` [16:18:57] inflatador: do you mean imbuing a service with an entirely independent discovery record, or something fancier? (e.g., something with a dependent depool on k8s-ingress-dse-aa) [16:19:36] (context) I've been looking at this separately for a different use case, and I think it may be techically feasible to support the former [16:19:54] emphasis on "may" :) [16:20:01] 👀 [16:20:56] swfrench-wmf: also seems like Sophroid becoming available could help with this kind of thing [16:21:12] yeah, since that would entirely side-step this :) [16:21:54] yeah I was thinking that too [16:22:02] there's also something really silly you could do [16:23:24] but let's figure out the actual constraints here first [16:24:10] how are your users reaching this service right now? a CNAME under .discovery.wmnet ? [16:25:44] (a bit more detail: I'm moderately confident that you can just slap a discovery service on a catalog-service that maps to the ingress IPs. basically, since state files are independent across services, it's fine that they share the same VIPs with the k8s-ingress one; same as when you create distinct discovery aliases) [16:27:10] swfrench-wmf: I've had it in the back of my mind that we could make the service catalog a lot less scary to deal with by being very prescriptive about where you must run the puppet-compiler at each step transition [16:27:37] :) [16:28:17] yeah, what gets tricky here is the referential integrity part w.r.t. conftool-data object definitions, but we could maybe come up with a way to mock that [16:29:54] yeah [16:30:17] as in, there's no explicit test-time constraint on that at the moment - adding the `discovery:` bits creates the intent to use them, but AFAIK we don't assert somehow that they've been created. same as how we can't / don't do that cross-repo with when the resources are added to the wmnet zone tempalte [16:30:22] right [16:30:49] but, +1 to scary-- and understandable++ [16:31:04] the "usual" approach there has been eventual consistency (in the loose sense) 'enforced' via an alert that fires if things are out of sync for too long [16:31:16] but ... for the service catalog there's plenty of states you just don't want to get into [16:31:38] * swfrench-wmf nods [16:32:33] I've been a little curious about revisiting the data structure there too, for something that's exactly as complex as it needs to be [16:32:51] doesn't touch the referential integrity aspects outside of that file obviously, but I've wondered if we can do better at "invalid states are inexpressible" [16:33:44] * swfrench-wmf thumbs up [16:33:46] (and maybe also at deduping, e.g. by treating ingress services as a first-class object -- yaml anchors are fine but we could make it simpler maybe?) [16:34:15] but all of that is in the "sure would be nice if we had some free engineer-hours" pile in the garage, and probably will be indefinitely :) [16:35:05] at least some of that would arguably be good next steps on Sophroid [16:35:10] yeah exactly [16:35:36] I'm tempted to go like, Sophroid first, with the data structure as she is today, and then start thinking about service catalog ng [16:35:54] sensible [16:35:57] but that's my "get sophroid out there" bias talking, and maybe it's the wrong order actually [16:36:15] (for one thing maybe it's easier to redo that data structure *before* adding a whole new consumer of it) [17:16:34] inflatador: the "how are your users reaching this service right now" was a question for you :) [17:26:30] cdanis sorry, was AFK. Reading scrollback now [17:30:22] So in this case, it would be a net-new service (WDQS replacement) running on dse-k8s. For AA services today, we CNAME them to `k8s-ingress-dse-aa.discovery.wmnet`, ref https://gerrit.wikimedia.org/r/c/operations/dns/+/1309183/3/templates/wmnet#1087 [17:31:05] But I don't think it's possible to failover individual services behind ingress this way(?) If I'm wrong LMK [17:37:53] no, but it's possible to have that CNAME point to something else :) [17:39:40] ACK, so does that mean we need to make DNS changes when we want to failover an individual service? [17:39:40] interjecting an unrelated question, sorry :) [17:39:43] btullis: admin-ng has some diffs for external-services, pointing at the new an-test hosts from T406746, is it okay for me to deploy those? [17:39:43] T406746: Ensure that our Hadoop stack works on bookworm - https://phabricator.wikimedia.org/T406746 [18:15:46] (I'm just going ahead with "-l name!=external-services" to deploy everything else and leave those dirty) [19:59:22] Low priority, but would anyone mind having a look at my cumin aliases patch? The expressions seem to work from the cumin host FWiW https://gerrit.wikimedia.org/r/c/operations/puppet/+/1309288 [21:27:34] ^^nm, got someone to look