[10:36:13] errand+lunch [13:16:16] o/ [13:37:47] \o [13:42:09] .o/ [13:44:05] o/ [13:56:39] Reminder that DC switchover is happening in around an hour, ref https://wikimedia.slack.com/archives/C016SNN9QQM/p1793021056374769 [13:56:55] hmm, should we worry about morelike in codfw? [14:00:08] there was a rapid switch last week as part of the codfw memcache loadtest and we did not react, we dropped some searches but sadly it's quite a thing these days... [14:01:42] I'm not sure. I know more memcached capacity was added to CODFW ( https://phabricator.wikimedia.org/T440896 ). [14:02:34] if it's a full week codfw only perhaps we should spread morelike to limit the annoyance on search users, someone just complained this morning about that (https://phabricator.wikimedia.org/T232612#12473174) [14:04:25] sounds like I need to bump the priority on T438238 [14:04:26] T438238: Cirrussearch: investigate unexpected rise in fulltext web searches - https://phabricator.wikimedia.org/T438238 [14:05:14] i don't think memcache capacity will help codfw, my analysis in spark showed the problem was that codfw in normal operation sees half as many unique titles daily to fill the cache [14:05:56] interesting [14:07:55] we dropped ~40 fulltext searches/sec last week during the memcached loadtest, we drop around 1/s in a multi-DC setup [14:08:58] Theoretically the lack of unique titles would take care of itself eventually once all traffic goes to CODFW, right? It's just a matter of how long we're willing to tolerate a degraded morelike experience? [14:09:23] Or are we burning up CODFW and affecting all search types until the cache is warmed up? [14:09:25] it's degrading mainly fulltext users not morelike itself [14:09:54] poolcounter settings for morelike is high enough that it almost never reject anything [14:10:31] hmm, i hadn't thought about that. indeed maybe morelike should have a lower limit [14:10:38] and yea, the cache rate should improve over a day [14:11:29] I do remember the CPU burning up in CODFW after the last switchover [14:11:31] wondering if plotting poolcounter rejections with a logscale makes it hard to read (or minimize actual bumps in rejections) [14:12:20] it seems to me that fulltext is the most important query type for users, but LMK if I'm wrong about that [14:13:04] https://grafana-rw.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?from=now-7d&to=now&timezone=utc&var-cluster=elasticsearch&var-exported_cluster=production-search&viewPanel=panel-65 [14:13:37] the bump at the beginning is the memcache loadtest where we ran 100% searches to codfw [14:22:52] so that load test appears to have doubled our fulltext P50 for about a day, raised the pool counter rejections and greatly increased CPU contention ( https://grafana.wikimedia.org/goto/s9l8pn?orgId=default ) [14:23:51] yes it's likely to be very similar when they switch today [14:26:38] And if we exempted morelike from the switchover, would that help the other search types enough to consider it? [14:27:47] what we usually do is send all the morelike traffic to alternate datacenter, which results in splitting enough load out that normal traffic doesn't get throttled [14:30:33] I guess we should do that then? Assuming it's not too late to get a MW config change pushed thru. Search doesn't have a formal SLO yet so it's a bit agonizing to decide (at least for me) [14:37:28] I stand corrected, looking at Asana it seems we do have a Search SLO now [14:37:41] https://grafana.wikimedia.org/goto/skg6jg?orgId=default [14:41:00] looks like the previous switchover affected the error budget, but not by much? LMK if I'm missing something [14:44:43] rejections are not included the SLO (according to T440163), they probably should I guess that's why it's not super visible [14:44:44] T440163: Count pool-counter rejections in the CirrusSearch query availability SLI - https://phabricator.wikimedia.org/T440163 [14:49:30] ah, good catch. I assumed it was covered by query availability [14:51:07] going to look into T442510 [14:51:07] T442510: cirrus-integ5.search VM busted, unable to finish booting - https://phabricator.wikimedia.org/T442510 [14:53:17] probably my fault... I remember I used the extra volume to put docker images... [15:00:28] How do we split morelike traffic between DCs? I thought that was in MW config somewhere but I'm not finding it [15:01:21] dcausse if there's important data on that VM LMK, I can probably help get it off [15:02:40] inflatador: the only thing I can't recover is the cindy private ssh key, it's possible Erik still has it somewhere if not we can probably set up a new gerrit account [15:02:52] we can always add a new ssh key [15:03:23] sure [15:03:52] actually i also have a ~/.ssh/id_rsa_cindy[.pub], so i probably have the old key still too [15:04:39] Let me know. It looks like it's just bailing out on a bad /etc/fstab entry, if WMCS supports rescue mode we could probably boot it into a blank OS and fix that [15:04:51] cool, I'll set up the machine and let you know when it's ready [15:05:29] inflatador: no big deal, I think it's fairly straightforward to set it up from scratch [15:05:40] dcausse ACK, I'll let y'all do that then [15:05:52] yea i think it's reasonably well documented, hopefully we didn't have many local hacks [15:24:16] Trey314159: do you have enough info to respond to the #talk-to-search thread? Basically asking us to provide some insight into recent work with typos [15:27:54] ebernhardson: I can reply. I think I was in the initial wish discussion when everyone else was on vacation one week! There's the second-try/DWIM stuff. Have we done anything else specific recently? (Everything feels all timey-wimey and "recent" has lost all meaning....) [15:28:45] Trey314159: hmm, not sure the timeline. looking [15:29:22] the page was created aug 2024, so probably not that far back :P [15:29:55] last updated march, reasonable to include everything since? Basically then i would also include reducing prefix-length from 2 to 1 on did-you-mean [15:30:06] T407432- [15:30:07] T407432: Follow-up AB test of dym language model variants - https://phabricator.wikimedia.org/T407432 [15:36:14] ebernhardson: when you have a moment could you put the cindy private keys to cirrus-integ6.search.eqiad1.wikimedia.cloud:/home/integration-bot/.ssh ? [15:36:47] ebernhardson: oh, yeah, that was good stuff! thanks for the reminder [15:38:10] dcausse: sure, sec [15:47:06] dcausse: should work, `ssh -p 29418 cindythebrowsertestbot@gerrit.wikimedia.org` connects [15:47:22] thanks! [16:17:25] hm... not sure what changed but ./mw dev mediawiki get-code pulled stuff into $PWD/'~'/git/gerrit/mediawiki [16:17:37] sounds like a bug with ~ [16:17:58] but it also pulled stuff into the usual place in the mediawiki subfolder [16:18:54] hmm, i probably set that manually at some point with the env? [16:19:29] hmm, it should have been done by this line in first-run.sh: $MW docker env set MEDIAWIKI_VOLUMES_CODE "$SCRIPT_DIR/mediawiki" [16:19:43] yes it should I see $MW docker env set MEDIAWIKI_VOLUMES_CODE "$SCRIPT_DIR/mediawiki" [16:19:53] somohow that got lost? [16:20:08] ./mw docker env get MEDIAWIKI_VOLUMES_CODE is now '~/git/gerrit/mediawiki/core' [16:20:12] hm... [16:20:15] weird :S [16:21:53] re-set docker env set again... [16:25:29] yes something is resetting MEDIAWIKI_VOLUMES_CODE to ~/git/gerrit/mediawiki/core [16:25:47] not sure what but possible mwcli itself, I pulled the latest version [16:26:14] sigh...sounds plausibly like an upstream bug? We could just expect it in the default location i guess [16:26:28] i only put it in the directory because i liked that it was all together, but it's not a big deal [16:26:37] yes [16:27:25] and ~/git/gerrit/mediawiki/core is not working the way it should, it's creating a folder named '~' in the current folder which I'm sure is not what was intended [16:27:46] sigh [16:28:03] use an older version of mw? I can copy one over [16:28:16] yes trying [16:42:08] well... it worked downgrading from 0.33.2 to 0.33.0... crossing fingers to not hit bugs fixed in these two patch versions... [16:53:54] already at 75% disk-usage and it needs to pull opensearch3 [16:54:26] dcausse what flavor of wmcs server are you using? I think there is an iops flavor with an extra disk but you have to format/mount it [16:54:52] or you can attach a Cinder disk, but same rules apply as far as format/mount [16:55:17] inflatador: yes... I did not use that one this time because I messed it up last time [16:55:32] I mean it's probably why it did not boot [16:56:24] was aware of cinder disk, might take a look [16:56:31] *not aware [16:56:35] yeah, I'd say attach a cinder drive and when you add it to /etc/fstab be sure to use the _nofail mount option [16:56:55] ah [16:56:58] that way if the disk is missing the server will still boot up [16:57:18] one of the most annoying things about Linux and storage ;P [16:58:01] yes... tho docker won't be happy without its /var/lib/docker/ but I guess the OS can still boot [16:59:18] ok cindy seems to vote again [17:00:33] can't seem to get the proxy to work properly tho https://cirrus-integration.wmcloud.org [17:00:49] curl localhost:8081 on the host works fine... [17:01:24] sorry, it's `nofail` WITHOUT the underscore, or `_netdev` with the underscore for the fstab option [17:01:36] either one works [17:10:43] created a volume, will try to attach it tomorrow, so far only the webproxy is not working as expected, unsure if it's me or still some limitation in wmcs [17:11:39] dinner [17:12:49] ebernhardson and others... I've replied to the talk-to-search Slack thread. Feel free to add or comment as needed. [17:14:06] yeah, I wonder if they shut off the webproxy due to recent security issues? [17:14:47] You might also try pulling the image using the .wmnet registry address? [17:20:15] Trey314159: thanks! [19:00:21] interesting finding, most growth from aug to now in full text search has referrer_class of none [19:00:58] grouped by (source, referer_class) then (web, none) it 64% of growth, (api,none) is 42% of growth, and a few splits have -% which balances it out to 100% [19:09:08] separately i'm impressed that taking the top 100 identities by query rate over the full time period, they mostly have UA's mathing the bot policy [19:10:17] i might have mucked something up though, the peak per-minute query rate comes from WDQS at just over 8k in one minute [19:15:01] Does `referrer_class=none` mean no HTTP referrer? And if so I guess that suggests a bot? [19:15:46] inflatador: yea that's the case. I don't know that it's a strong enough signal on it's own though [19:16:37] but for example grouping by UA, the largest non-referrer ua is WikipediaApp/6169-r-2026-09-04 (iOS 26.6.2; Phone) [19:18:15] I guess that makes sense [19:19:30] I guess if most of the UAs are compliant, that means we're getting more bot traffic, but it's well-behaved bot traffic? [19:20:27] well, the bot's could be rotating UA's, it's hard to say :S. I'm trying a few different ways to get some ideas [20:06:27] curiously, bingbot seems to do a lot of queries [20:07:06] by asn, microsoft (maybe includes azure?) has ~1.9M queries from 176 ip's for one day [20:08:05] interesting [20:08:40] If it's bingbot, I assume it's the crawler as opposed to referring human search traffic? [20:14:29] yea. That's only a subset, but it is a pretty big part of automated traffic [20:14:50] 2M queries per day is still only ~25qps [20:20:53] i have found what plausibly looks like a set of ~10 UA's that come from residental networks and don't look like "typical" UA's. high zero result rate, referrer rate of 0, cookie rate of 95% when "normal" mobile traffic is seeing ~50% [20:23:33] there might also be real growth in tools, i wonder if cloud is seeing increased rates of apps deployed with AI helping people [20:34:12] That's a weird one, what does cookie rate suggest? I would've assumed no cookies meant a bot [20:34:30] or at least more likely to be one [21:01:40] cookie rate is the percentage of requests that have at least one cookie (i had expected high, since we set cookies for last-access) [21:19:34] picking up from our discussion this morning, do you think we have any headroom to increase the pool counter for search? I know we are burning up the CPUs in CODFW, but I'm not seeing any thread pool rejections on individual cirrus hosts