[13:17:35] o/ [13:51:55] \o [13:53:06] o/ [15:09:38] huh, finding while digging through these opensearch bits that it now does concurrent segment search under certain circumstances. It existed in 2.x, but they switched the default in 3.x to 'auto' which uses it. Sadly it looks to mostly be for aggregations. [15:49:39] hard to tell what that'd mean for our workload if normal searches were to use more than one active threads per searches [15:50:40] somewhat related just saw that there's an async search plugin, not totally clear if this could be useful, the book-keeping part could be annoying/dangerous I guess [16:20:06] yea it seems like trading latency for throughput. really hard to say. [16:20:41] for async, i guess that seems more useful for things that run for awhile. not clear it would help us much [16:22:25] yes definitely [16:24:36] seems like we switched search to 100% codfw? [16:29:14] happened several hours ago... just noticed it now because latencies are not great [16:30:10] dcausse yikes, I guess I'll go check the SAL and see what might've prompted that [16:32:03] looks like T440896 [16:32:03] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [16:32:21] yes seems like it thanks [17:08:35] should we ask for a cache warm-up before depooling under those circumstances? I know ebernhardson has said it doesn't do much for morelike though [17:11:06] inflatador: yea i don't think cache warm-up does anything right now. It did something when we went from 0-traffic to all-traffic, but since both dc's serve traffic the opensearch level caches are reasonably full [17:11:50] with no search traffic it would be only doing writes, it wouldn't have the older, larger segments in disk cache [17:14:14] ACK, good to know [17:14:49] sounds like they got more memcached machines in CODFW at least. I don't wanna know how big those caches need to be ;P [19:02:49] restarting CODFW to apply some kernel updates