[08:09:57] ebernhardson, dcausse, Trey314159: you should have received (or will receive soon) an amail about getting a yubikey. This will be needed to SSH into our servers starting November 5. Please go ahead and email techsupport@wm.o as soon as you can to order your key (include address and phone number). There are options for USB-A or USB-C keys, so give your preferences. [08:10:59] DPE SRE is there to help configure them if you need, but the process is simple enough, I'm pretty sure you'll figure it out (https://wikitech.wikimedia.org/wiki/Yubikey-SSH). [08:11:24] There's also a slack channel to get help (#hardware-ssh-key-support). [08:11:53] Sorry for the rush, I know this is a short timeline. [08:22:48] gehel: thanks [11:21:59] lunch [13:00:34] o/ [13:40:04] \o [14:05:25] .o/ [15:37:44] should we move morelike back to eqiad? we are dropping even more today than yesterday [15:41:04] +1 [15:42:33] +1, how do we do that? I was looking at mwconfig and didn't see anything obvious [15:43:49] inflatador: yea it's not obvious, we use: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1360455 [15:44:15] LOL, silly me was looking for `morelike` instead of `more_like` [15:44:25] yea...we aren't consistent there [15:45:00] No worries, I should have been able to find it anyway [15:49:18] inflatador: these are the UA's from the bots i identified, i don't know if it will be effective to just throttle these though they will plausibly simply change UA's: https://phabricator.wikimedia.org/P97026 [15:49:26] those are all the same bot afaict [15:51:00] ebernhardson ACK, will take a look. We have other ways of finding those bots. You're probably already aware of 'em but happy to discuss in a non-public channel if you're curious [15:54:34] this is probably a stupid question, but what are you using to ID these as searches? Like, URI pattern of https://${wiki}/w/index.php?search=${foo} ? [15:56:31] inflatador: that one was grouping by user agent from cirrus request logs and looking over the top 100 agents along with a variety of bot signals like did they send cookies/accept header/frequency of zero result queries/unique query ratio/etc. [15:57:37] basically those UA's are near the top of the list, the 10 UA's request rates move almost in lockstep as if centrally controlled, and their metrics are far off what other mobile ua's report [16:02:47] poolcounter graph declining, down to 110. but it averages over a few minutes. Will keep watching [16:04:50] thanks! [16:06:54] +1, thanks! I'll make an alert for the rejections if we don't have one already. Heading to lunch first though [16:46:31] declined, but not all the way. For some reason codfw can't run the whole full-text load right now. But curiously we know eqiad, with the same size cluster, ran the full query load acceptably when codfw lost power [16:58:09] looking into the difference between clusters...i dunno. The short answer is when eqiad served the full load cpu usage wasn't quite as high as we are seeing in codfw right now [17:01:51] but why? i dunno... [17:02:54] the clusters have a mix of cpu's at 2.1Ghz or 2.4Ghz depending. haven't looked at the exact mix, but i imagine a small difference in the balance shouldn't have a big enough effect [17:11:01] back [17:12:23] when eqiad had full load we peaked at ~2.7k overall qps, we are seeing slightly higher peak of 3k today, but with more_like moved we are down at a similar 2.7k [17:14:46] I'll take a closer look. There's a possibly-relevant mainline SRE tickets around the switchover T438896 and a claude-aided investigation https://people.wikimedia.org/~cdanis/2026-switchover-overload/ [17:14:47] T438896: Investigate high saturation in codfw - https://phabricator.wikimedia.org/T438896 [17:18:04] T441436 also talks about a system for setting weights between DCs, so we could do something like a 60/40 split of a service instead of all or nothing. It wouldn't work for query types, but still interesting [17:18:05] T441436: Trial of weighted dnsdisc for mw-web-ro switchover (gradual ro traffic shift) - https://phabricator.wikimedia.org/T441436 [17:18:47] hmm, yea that would plausibly help. This is the first time the moving more_like hasn't been enough to fix the problem, we will need a bigger hammer [17:19:30] i'm not sure it's worth the effort, but have pondered introducing a dnsdisc for morelike so we could move those with sre tooling. If we can move a fraction of all traffic that's probably good enough [17:20:27] yeah, I guess there's still quite a bit of SRE-type work to do re: performance differences between DCs. I'll get a ticket started for that too [17:34:44] I think know why the automated poolcounter is no longer working, the header we used got replaced by another one (T396621) [17:34:45] T396621: Requestctl should use x-provenance header - https://phabricator.wikimedia.org/T396621 [17:35:40] will update the headers we collect in the logs that should help revise the pool counter classification [17:37:17] and we should probably move all these classification bits to the private repo I'm afraid... [17:37:19] dinner [17:59:10] dcausse: btw there's actually kinda docs for that now https://wikitech.wikimedia.org/wiki/CDN/Headers#X-Provenance [18:09:34] random thought, if we moved more_like traffic and search threadpool usage dropped from 3k to 2.4k threads, but we are still rejecting queries at the pool counter level, that suggests our limits don't line up well [18:10:22] we are still lightly queueing on the opensearch side too though...always hard to say [18:12:02] yeah, I was thinking that too...even though there is a lot of CPU contention on the CODFW cluster, none of the individual nodes seem to be giving thread pool rejections. Maybe try bumping up pool counter until we start to get some? [18:13:10] yea it might help, it's annoying that it takes a mw deploy to change those things. [18:16:56] Yeah, maybe it is worth exposing query types at some point. My dream would be to migrate us to Envoy and use https://wikitech.wikimedia.org/wiki/Sophroid instead of discovery. [18:18:49] That's not quite ready for production, but I like the idea of not having to expose services in discovery to get load balancing/service discovery features [18:19:56] huh, that's interesting [18:24:18] at my old job we used Consul for a lot of this stuff. We're kinda playing catchup with a homebrew approach, but it does look promising. And it's not like Consul was great at programming Envoys either ;P [19:10:06] https://grafana.wikimedia.org/goto/sgsgrd?orgId=default I updated the pool counter rejections/second panel to include the DC, if that's not useful feel free to revert or I can split out different panels [19:12:15] yea that seems reasonable [19:19:43] Also created T442912 to look at DC performance differences and try a few things. Feel free to add suggestions/rebuttals etc [19:19:44] T442912: Cirrussearch: Investigate performance difference between EQIAD and CODFW - https://phabricator.wikimedia.org/T442912 [19:32:06] Ah, and it looks like we already have a pool counter rejection alert: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-search-platform/cirrussearch_k8s.yaml#6 . Currently fires at average rejection rate of 1000/5m after 1 minute, maybe we should drop down to 200 or so? https://w.wiki/fJfa shows we peaked at around 280 earlier today [19:38:56] it looks like we created that in T262694 when we had 3.1k/s pool counter rejections that didn't alert. I guess in that context 1k seemed appropriate [19:38:57] T262694: Search is currently too busy - https://phabricator.wikimedia.org/T262694 [19:39:30] hmm, i agree that 200 is a significant event either way. But perhaps we make it wait for 10 minutes of > 200? [19:39:40] or 5 min, i dunno. I guess it's already an average over 5 minutes [19:44:50] with some reading, i guess the alertmanager `for:` statement isn't well liked and should stay low, but it's already an average over 5m so probably fine [19:48:54] Yeah, I'm flexible on the number...the thought is just that if it's a big enough problem to change DCs it's probably big enough to alert [19:49:54] I'll do a patch for 10m of 200 to start with [19:56:48] +1 [20:44:46] https://gerrit.wikimedia.org/r/c/operations/alerts/+/1360534 as promised! [21:15:24] ebernhardson sorry to bug again on the UA thing, are you looking at cirrus request logs with a jupyter notebook or how do you typically do it? [21:17:41] nm, found it