[00:52:18] <inflatador>	 Unless I'm reading it wrong, we're already 25% done with the snap?
[06:26:20] <dcausse>	 we could try dumping json data from codfw and push through classic _bulk ingestion, should work and workaround versioning issues
[06:40:00] <dcausse>	 actually it's "only" 4.4M docs to restore, could possibly write a small script that diff the two cluster and force-import the missing files
[08:00:00] <dcausse>	 started to dump commonswiki_file page_ids, I seem to be able to filter on _shards:9 which should speedup the comparison
[08:10:46] <dcausse>	 ok started to restore the pages
[08:20:11] <dcausse>	 sigh... a bit slower than I would have expected... current progress estimates are around 7days to restore this shard :/
[08:22:46] <dcausse>	 ah no (estimated 44M pages, it's actually 4.4M), down to 17h to restore this shard
[08:33:28] <atsukoito>	 good morning! reading thru log before starting working on eqiad
[08:36:37] <dcausse>	 hi!
[08:59:09] <atsukoito>	 dcausse: about the lost shard, do you want to run the sync script now, or should we try restoring from snapshot and then syncing the leftovers?
[09:00:08] <dcausse>	 atsukoito: you can try importing the snapshot into a new index, but I suspect it'll fail because we still run opensearch 1.x nodes
[09:00:46] <dcausse>	 if the snapshot succeeds I'll cancel my script and start running a backfill
[09:03:49] <dcausse>	 but with only 16nodes with opensearch2 I doubt it's going to work but possibly worth a try
[09:06:00] <dcausse>	 atsukoito: otherwise we could continue the rollout and as we get more opensearch2 nodes we can retake a fresh snapshot and import it if the sync script is not done at this time
[09:08:29] <atsukoito>	 i think i'll continue rollout  for now, and then we figure it out. we could continue script running
[09:11:06] <dcausse>	 sure
[10:16:07] <dcausse>	 lunch
[10:47:12] <atsukoito>	 lunch, too
[12:27:15] <inflatador>	 <o/
[12:39:15] <dcausse>	 o/
[12:58:00] <dcausse>	 sigh... my script re-indexed from the sane cluster back to the same sane cluster... restarting
[13:10:53] <inflatador>	 I think there are 32 shards in that index, we could try a restore once we get to that number of OpenSearch 2 hosts
[13:13:09] <dcausse>	 sure, we can also finish the upgrade, there's no real rush actually
[13:14:39] <dcausse>	 ok, restarted the sync script (to the right cluster this time...), if this is done before the end of the migration there'll be no need to restore a snapshot
[13:17:16] <inflatador>	 Awesome, thanks for the help. And yeah, if it doesn't work we can finish the migration and restore afterwards
[13:18:24] <dcausse>	 lost around 5hours with my mistake, based on previous estimates it should be done tomorrow around 7 or 8 am UTC
[13:20:08] <inflatador>	 Is there a way we can check progress?
[13:22:47] <dcausse>	 inflatador: sure, running "curl -s https://search.svc.eqiad.wmnet:9243/commonswiki_file/_count?preference=_shards:9 | jq .count" should progress to a value close to what you see in codfw (~45M)
[13:23:20] <dcausse>	 249128 currently
[13:31:43] <ebernhardson>	 \o
[13:38:48] <ebernhardson>	 dcausse: thanks! i had pondered that but didn't occur to me to pull from codfw for some reason, makes sense
[13:39:00] <dcausse>	 o/
[13:42:22] <dcausse>	 it's kind of slow tho, I suspect the snapshot restore is much faster, would be great if it could restore a single shard into an existing index
[13:42:55] <ebernhardson>	 yea i was kinda surprised to see no single-shard restore.  I kinda understand why, but it would be a nice feature
[13:51:19] <ebernhardson>	 hmm, we are a top hdfs user in unexpected place: 29.3T /user/analytics-search
[13:54:57] <dcausse>	 sigh...
[13:55:09] <ebernhardson>	 turns out, it's .Trash.  That's basically what we delete in 30 days
[13:55:20] <dcausse>	 ouch :/
[13:55:24] <ebernhardson>	 i guess the question is, how do we delete more data than anyone else?
[13:56:07] <dcausse>	 I did a mass delete recently but I thought it was done more than 30days ago?
[13:56:54] <ebernhardson>	 i think it's the cirrus dumps getting weekly deleted, to the tune of 4-5TB/wk
[13:58:20] <ebernhardson>	 it's been running long enough and we've never needed a restore, going to add --skipTrash to the data cleanup commands
[13:58:57] <ebernhardson>	 (prompted by a general hdfs cleanup request from dpe for hadoop)
[14:00:55] <dcausse>	 sure
[14:03:08] <dcausse>	 ah seems like /wmf/data/discovery/wikidata/rdf_subgraphs/ is not clean up properly :/
[14:03:25] <dcausse>	 34T at 400G/week
[14:04:07] <ebernhardson>	 hmm, probably also worth looking at.  We aren't the worst users (i think), but our home dir was the second largest on the cluster at 30Tb
[14:04:35] <ebernhardson>	 /wmf/data/discovery reports just under 100TB, more than i expected i guess
[14:04:55] <dcausse>	 a third of this is wikidata related
[14:05:44] <dcausse>	 will file a ticket
[14:06:18] <dcausse>	 same issue as always I guess, wrong snapshot partitioning with YYYYMMDD format :(
[14:07:04] <ebernhardson>	 according to claude, old hadoop (us) reports logical size,  so full size is *3 replicas as well
[14:07:15] <ebernhardson>	 new hadoop apparently prints both, would be nice
[15:18:42] <dcausse>	 ok /wmf/data/discovery is down to 60Tb, I don't see anything obvious to reclaim
[15:19:32] <dcausse>	 could possibly use alternative compression formats for things that are not read-heavy like opensearch semanticsearch dumps
[15:24:10] <ebernhardson>	 thanks!
[15:26:37] <pfischer>	 o/ Just got out a design walkthrough for semantic search phase II. Tais asked, if there’s any chance to get more focused snippets (= highlights within the snippets) from semantic search. IIRC, we already experimented with the semantic highlighter built into OpenSearch, but the results weren’t great. Was that based on the embeddings at paragraph level already?
[15:27:48] <dcausse>	 pfischer: semantic highlighting is actually just a reranker ran on every sentences, it'll probably be extremely costly
[15:28:20] <dcausse>	 and yes the one provided by opensearch was pretty weak imo
[15:29:51] <dcausse>	 and super slow
[15:32:15] <ebernhardson>	 we could probably look into it, but indeed i was very un-impressed by the built-in highlighter
[15:34:15] <dcausse>	 and yes it was already based on paragraph level chunking
[16:57:10] <dcausse>	 dinner
[19:38:37] <inflatador>	 looks like the shard restoration is at about 98%, based on d-causse curl above vs. codfw shard
[19:46:38] <ebernhardson>	 nice! Partition 9 looks a little small, but i suppose the deleted docs count is going to be very low there. Its at ~55gb vs ~80gb for other shards
[19:47:18] <ebernhardson>	 i guess 70-85gb, the shards vary in size more than i would expect considering its commonswiki, but doesn't matter
[20:04:12] <inflatador>	 we're back in red again ;( I'm gonna be able to get the host back this time though
[20:04:51] <ebernhardson>	 do we need to re-visit how hosts are selected?
[20:05:18] <ebernhardson>	 once could be a fluke, but twice suggests we need to understand what host selection is leading to all valid shards being gone
[20:05:28] <inflatador>	 oops, never mind
[20:05:35] <inflatador>	 just me misreading
[20:05:38] <ebernhardson>	 ok :)
[20:06:39] <inflatador>	 and yes, we do need to revisit how hosts are selected. We really need to tie the reimage process back into rolling-operation, but there's an impedence mismatch there
[20:07:51] <inflatador>	 reimages work about 80% of the time, and whenever it would fail, we'd have to fix a bunch of half-broken stuff and then manually pick up where we left off
[20:08:11] <ebernhardson>	 hmm, that does sounds like a pretty meh process :S
[20:09:33] <inflatador>	 I think we'll be able to do it eventually, if we could put cookbooks behind airflow and have it keep track of success/fail and retry for us