[10:11:42] lunch [13:08:55] o\ [13:15:35] hi! [13:16:04] dcausse looks like the shard recovery is finished based on size, does it look OK to you? [13:16:11] o/ [13:16:24] ref https://wikimedia.slack.com/archives/C055QGPTC69/p1783487436666359?thread_ts=1783343349.767329&cid=C055QGPTC69 [13:17:07] inflatador: I believe so, doc count is similar, checked a few docs and they have identical sizes [13:19:19] excellent, thanks! I'm gonna add a backup/restore section to https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration [13:22:15] should we have a quick script that checks if the host is safe to be re-imaged, IIRC we can't ensure green before starting the next host but removing a host while being yellow is prone to primary shard loss but quick script could tell us whether it's safe or not [13:22:50] seems reasonable, i was thinking it shouldn't be too hard to do some set math on a few api responses to decide if a host can go [13:23:06] or a set of hosts perhaps [13:23:46] yes, I think we could probably borrow the logic from rolling-operation. Per scrollback we can't use rolling-operation for reimages [13:24:36] i mean this could be a bash or python script, just something like 'allow-reboot.sh && restart' [13:24:49] or however the instances actually get rebooted [13:24:59] or shutdown i guess [13:25:16] Another thing we might consider is making partman recipes that ignore the data drive [13:25:23] yes... I think I had one, lemme check [13:25:46] dcausse: i'm working on figuring out this part (is it sfe to pull it down) here https://gitlab.wikimedia.org/repos/search-platform/sre/es-tools# [13:26:02] atsukoito: thanks! [13:26:59] very nice! [13:28:21] yes I have a very dirty shell script named are_we_safe.sh, it just uses the set of hosts from .persistent.cluster.routing.allocation.exclude._name (assuming that's the one we want to restart) and check whether they're hosting shards that are not assigned anywhere else [13:28:47] Now I can say for sure when there are active migrations, and I'm working on checking all the shards from `_cat/shards | grep -v STARTED` against `_cluster/state` [13:28:53] I usually go with the working assumption that we can do 3 at a time, as long as they're in the same row. Problem was my inattention when switching between rows [13:29:15] So yes, anytime I can get a computer to pay attention for me, it's a win ;) [13:29:26] same :) [13:29:28] this will show if there are any hosts that are holding a single copy of the shard (maybe I don't even need `_cat/shards` tho) [13:29:46] atsukoito: i dunno if it's easier, but it's non-obvious that the _cat api's can respond with json instead of whitespace by sending &fomat=json [13:29:56] usually easier to automate with json [13:30:27] thanks ebernhardson, i will update it (because I'm now parsing it as text :<) [13:30:45] They can also return YAML, not sure if it's useful here but just FYI [13:47:43] I'm trying to think through whether or not retaining the OpenSearch datadir on a reimage would help avoid data loss [13:47:58] inflatador: seems like it could only help [13:48:19] It takes about 45m to reinstall, so the rest of the cluster would give up on the shards and self-heal [13:48:34] yea by then the shards wouldn't be needed, but if it goes red we have the data [13:48:47] then the reimaged node comes up with old versions of the shard, would OpenSearch handle that OK? [13:48:48] but as this showed, a single shard isn't that big of a deal. Maybe better recovery automation? [13:49:12] Yes, that would help too [13:49:45] But in the case where we did lose the shard, index goes read only, do we still have data loss, or just backpressure from the updater? [13:50:10] Like if we lose a shard, but it comes back in 45 minutes. I imagine OpenSearch would consider that shard too old and we might have to manually tell it to accept old data? [13:50:35] depends how the docuemnts come back. In this case david copied between clusters live. It could have missed an update i guess if it raced, but mostly stayed up to date [13:50:43] the index should stay red and all writes would fail [13:51:00] oh, yea if it's all shards failing writes...then SUP needs to run a backfill [13:53:55] So even on a red index, I guess OpenSearch only blocks writes on the particular missing shard? [13:54:27] hmm, i suspect (untested) it blocks all wrtes [13:54:34] could be shard level though [13:54:39] yeah, that is my understanding too [13:54:46] but I'm not sure [13:55:11] Does that mean we need to do a backfill for the index with the lost shard? [13:55:32] i suppose yes, it does suggest we missed a few edits (that saneitizer will clean up ove a week, hopefully) [13:55:37] s/a week/two weeks/ [13:56:34] ACK, so no manual backfill needed [13:56:58] i suppose the thing to remember that saneitizer doesn't do is fix up weighted_tags [13:57:06] shouldn't matter in commonswiki though [13:57:36] possibly silly question, but why not? [13:57:37] there isn't really a canonical weighted tags store anywhere (or there wasn't in the past, unclear today) so no reasonable way to correct them. Just wait for another edit [13:57:56] on another edit the ml based tags get re-sent [13:58:28] or perhaps re-classified is the appropriate description [13:59:08] yes, would be nice to find a solution for that but that seems tricky, esp. because weighted tags was designed to allow others to push, not cirrus to know where to pull data [13:59:19] yea [14:51:36] ah, looks like P&T mtg is at the same time as Weds mtg, I'm gonna go to the P&T [14:51:48] o/ What’s the mood regarding P & T staff meeting? Would you want to attend live or shall we have our Wednesday meeting [14:53:10] I'm ok with going to P&T and then picking up the Wednesday meeting afterwards for those who can attend [14:53:19] yea thats fine [14:56:46] I'll have to get my car from the garage around in 15mins so happy to join the wed meeting after that [15:03:15] Alright, I won’t be able to join after P&T but if you could look into options for improving the highlights of semantic search results, that would help a lot. Apparently there was a shift in priorities so routing shall only be applied, once we have a reliable, cross-language, data-backed heuristic from Research. The “Find” button will always fetch semantic search results. [15:14:36] Any objections to me deleting the commons_file snapshot I created? Just trying to quash an alert [15:19:21] errand [15:19:22] inflatador: no objection [15:24:59] ebernhardson ACK, will get on it [16:13:56] I asked claude (with access to CirrusSearch and extra) to propose options for highlighting: https://phabricator.wikimedia.org/P94769 [16:29:16] pfischer: do you know if the ML team machines you told us we could use are still available? [16:54:49] dcausse do you know which machine you wanted? I know btullis was gonna use some of 'em [17:04:50] inflatador: I don't really know I remember there were 7 machines [17:15:34] dcausse if it was these guys, you might be out of luck :( T430654 [17:15:35] T430654: decommission ml-cache* - https://phabricator.wikimedia.org/T430654 [17:15:52] If relforge would be useful instead LMK, it's empty now [18:25:25] https://gitlab.wikimedia.org/repos/search-platform/sre/es-tools now it shows if the data is transferred or, if it is not scheduled yet, what hosts can potentially be used as sources [19:39:52] an example of highlighting from passages with ML. Given the opening text to Eiffel Tower, split with only newlines, it identifies the '330 meters' span as the answer to 'How tall is the Eiffel Tower?'. If asking the same passage 'How tall is the Champ de Mars?' it gives no-answer. https://phabricator.wikimedia.org/P94770 [19:40:31] quite slow though, it was about 60s of cpu time to load the model and give no-answer [19:45:11] Given the same english question but the opening text from frwiki Tour Eiffel, it identifies 330m as well (even though the text refers to multiple periods where it was shorter) [19:45:51] Is that just thru OpenSearch or are you using liftwing too? (Sorry if dumb question) [19:46:24] this is just pytorch on cpu with a model from huggingface. Given the speed i'm seeing, it would probably have to be deployed on liftwing (if its fast enough there) [19:47:14] ah OK, so there is no OpenSearch in the loop at all yet [19:47:42] no, we would have to ponder what we want to do there. I suspect the POC implementation is cirrussearch extracts the passages from the hits and passes them to an api (like liftwing) [19:48:08] we might be able to integrate to the opensearch highlighting, but have to evaluate what the api looks like for making custom remote calls [19:48:33] but it should be possible [20:13:43] yea very slow...with 3 paragraphs 85s to cpu infer with large model, 40s w/ base model, 6s with deepset/minilm-uncased-squad2, but it found 300 m instead of 330 (initialement nommée « tour de 300 mètres ») [20:24:30] i guess minilm is on bert, and deberta large is a couple generations newer [20:31:36] reading through, shouldnt be too bad. looks like integrating a remote highlight model in opensearch is very similar to a reranker which we already played with [20:39:36] commonswiki_file shard 21 is down to a single shard, doesn't seem to want to self-heal even after a reroute. hmm [20:41:17] :S [20:44:43] so it says "temporarily throttled". Does that mean there are too many ongoing recoveries and it's waiting for a spot? [20:44:54] basically the current capacity to recover shards is maxed out [20:45:22] there are 38 active recoveries at the moment, seems plausible [20:49:28] yeah [20:49:44] i suppose the question might be, can we make recoveries process faster? or allow more in parallel? I'm not sure but poking around [20:50:04] Yeah, I've turned it up higher before, sometimes it causes issues with the cluster state though [20:50:25] usually it's at 4 per node, I have it bumped up to 7 for the migrations [20:53:44] I think it's OK to wait it out in this case [20:54:54] yea, i suppose maybe i'm surprised it's not spread out more. 53 nodes, 7 per node gives theoretically 371 concurrent recoveries. We have 37 active recoveries. Not that i expect it to get anywhere close to 371, but 10% also sounds low [20:55:36] inflatador: maybe node_concurrent_outgoing_recoveries? [20:56:18] oh nm, the one we set should set both [20:56:53] I think we have anti-affinity on these large shards too, at least that's what I'm seeing from `/allocation/explain` [20:57:05] yea probably there just aren't enough choices [20:59:38] i suppose what seems wrong here sis some commonswiki recoveries hve been running for 3h [20:59:50] wow i can't type... [21:00:02] Also seeing occasionally 503s from the API, that might mean we have a bad node in LB rotation or too much cluster state to process ATM [21:00:12] i mean the commonswiki_file recovery on shards 23, 24, 28, and 29 that report recovering for 3 hours feels too long [21:00:49] we probably have to lift up the io limits? not seeing them in the _cluster/settings [21:01:43] indices.recovery.max_bytes_per_sec [21:04:15] interesting, let me check [21:04:21] oh nifty, you can pass `?include_defaults=true&flat_settings=true` to /_cluster/settings to get a format that is more greppable [21:05:05] so something like this finds the effective config: curl -s 'https://search.svc.eqiad.wmnet:9243/_cluster/settings?include_defaults=true&flat_settings=true&pretty' | grep indices.recovery.max [21:06:29] nice [21:07:30] i dunno whats safe, i would perhaps want to set the speed based on some sort of requirement, say it should be able to transfer our largest node in < 1hr? [21:08:21] 50GB in 1 hour would be ~ 14MB/s, with 7 parallels thats 100MB/s cap. [21:08:47] in theory the 10G nics do >1000MB/s [21:09:35] dunno what our write throughput would be, i guess it might be disk writes/reads and not Nic [21:09:43] yeah, that's what I'm looking at now https://grafana.wikimedia.org/d/popinski/node-exporter-cirrussearch-hosts?orgId=1&from=now-7d&to=now&timezone=browser&var-DS_PROMETHEUS=000000005&var-job=node&var-nodename=cirrussearch1068&var-node=cirrussearch1068:9100&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B&refresh=1m&viewPanel=panel-340 [21:09:59] I have a sorta half-baked dashboard I'm working on based on the PSI metrics [21:10:29] basically to see how I/O bound we are already [21:10:58] oh nice [21:11:38] PSI metrics are great, takes a lot of the guess work out of whether stuff is IO/CPU/mem bound [21:12:20] in this case we are already hitting some pretty nasty I/O spikes, but they're always really brief [21:12:30] Dunno if that's sample rate being weird or something [21:17:03] did some reading of the kernel docs for PIA, 1% full seems not great but also not bad. [21:17:23] I suppose in theory, as long as the streaming updater isn't getting backlogged the stalls are "fine" [21:17:48] at some point the throughput declines as PSI rises, but i'm not sure where. probably have to find it experimentally [21:18:06] s/PIA/PSI/ [21:34:23] We have alerts for I/O stalls over 10% for 5m, that's from when we reimaged cloudelastic and it didn't have working readahead yet. But I fudged on that, wasn't really based on anything [21:39:00] 10% makes sense, if 10% of cpu time is spent with everything stalled on io that's pretty high [22:54:49] Aannnnd, we are all finished in EQIAD