[09:47:27] <atsukoito>	 tl;dr I dropped this alert and will be looking how to resume uploading logs to logstash, the log messages are much more actionable than the counters
[09:50:50] <atsukoito>	 dcausse: hi!
[12:35:45] <inflatador>	 <o/
[12:45:37] <atsukoito>	 inflatador: hiya!
[12:47:36] <inflatador>	 Greetings from https://en.wikipedia.org/wiki/Galveston%2C_Texas !
[13:14:07] <ebernhardson>	 \o
[13:16:43] <inflatador>	 .o/
[13:19:09] <ebernhardson>	 hmm, kbdwiktionary_content failed 4 reindex attempts: "cause":{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [indices:data\/write\/bulk[s]] would be [10260384974\/9.5gb], which is larger than the limit of [10200547328\/9.5gb], real usage: [10256284720\/9.5gb], new bytes reserved: [4100254\/3.9mb], usages [request=0\/0b, fielddata=4044797\/3.8mb,
[13:19:10] <ebernhardson>	 in_flight_requests=4100254\/3.9mb]","bytes_wanted":10260384974,"bytes_limit":10200547328,"durability":"TRANSIENT"}
[13:19:58] <ebernhardson>	 10g is massive for in-flight bulk writes?
[13:21:56] <ebernhardson>	 oh, reading it looks like thats in "real mode" which means 10g is just the total heap, not whats allocated to that breaker
[13:31:41] <ebernhardson>	 eqiad and cloudelatic are *still* reindexing commonswiki :P
[13:32:38] <inflatador>	 LOL, wow
[13:33:08] <inflatador>	 Do we know how long CODFW took?
[13:33:12] <ebernhardson>	 yea it didn't used to be that bad...i have a vague memory that when we wrote the reindex orchestration the whole cluster took 4-5 days
[13:33:48] <ebernhardson>	 hmm, i didn't actually include any info about finished reindexes in the reindex report :S
[13:35:05] <ebernhardson>	 looks like from the raw timestamps in the state log, 47.87hrs
[13:35:33] <inflatador>	 Interesting, I wonder what the difference is?
[13:35:54] <ebernhardson>	 i don't know if it's true, but i get a vague feeling every time software gets "upgraded" it also gets slower :P
[13:36:15] <ebernhardson>	 could also be changes over time, additional fields we add, increasing the per-doc indexing cost
[13:39:38] <ebernhardson>	 i wonder how we could better quantify that...would almost need a default set of docs we push through a standard pipeline. But doing that in CI seems rough, it gets assigned to random machines with different hardware
[13:44:05] <inflatador>	 But CODFW's reindex was also on OpenSearch 2?
[13:46:01] <ebernhardson>	 yes, it just restarted more times
[13:46:53] <ebernhardson>	 eqiad's current reindex has been running for ~36hr
[13:47:01] <ebernhardson>	 codfw finished in ~48
[13:58:35] <inflatador>	 But that's the entire reindex for CODFW, vs just commonswiki for EQIAD/cloudelastic?
[13:58:56] <ebernhardson>	 no, 48h is just how long it took to do commonswiki_file in codfw
[14:00:05] <inflatador>	 ACK, so I guess none of 'em have completely finished a reindex on OpenSearch 2 yet?
[14:00:53] <ebernhardson>	 codfw completed ~123 indices (after commonswiki_file finished it then does more in parallel), but cloudelastic and eqiad are still working on commonswiki_file
[14:01:15] <ebernhardson>	 basically the way i configured the limits it will do commonswiki_file with no other parallelism
[14:02:08] <ebernhardson>	 it's not the terrible since we don't have to really hand-monitor it, but i just worry a bit about maint operations that take long time
[16:04:41] <ebernhardson>	 huh, so checking codfw tripped circuit breakers, it's almost entirely one host (2078).  166k tripped on 2078,  next closest is 2066 with ~800 trip's
[16:05:41] <ebernhardson>	 but it's only 4 hosts in codfw and 3 in eqiad (for the big cluster) that tripped breakers, out of 50+ nodes per cluster.  So again it feels a bit like a luck of the draw with shard assignment :S
[16:25:40] <atsukoito>	 inflatador: I'm switching the healthcheck to `/_cluster/health?timeout=5s`, this way it won't eat connections (PyBal is timing out at 5s anyways) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1310129
[16:25:46] <atsukoito>	 will roll it tomorrow morning
[16:28:17] <inflatador>	 atsukoito ACK, that makes sense. Will take a look once I get back from lunch
[16:49:57] * atsukoito signing off for today
[17:50:37] * ebernhardson sighs... CI failed because https://packages.adoptium.net/artifactory/deb/dists/bookworm/InRelease is returning 503's...