[08:22:09] good morning! [08:29:44] dcausse: i checked that the long queries are working. did you do `indices.query.bool.max_clause_count` manually on the cluster? or it was picked up from the config? [08:29:53] re: T431086 [08:29:54] T431086: Searching with wildcards fail with OpenSearch 2 - https://phabricator.wikimedia.org/T431086 [08:38:59] hi! [08:39:41] atsukoito: I did not set anything in eqiad [08:40:03] cool, then the setting was picked up from the config! [08:40:23] nice! [08:41:49] I was figuring out how to track down hosts that we shouldn't power down yesterday and... OpenSearch is soo annoying.. [08:43:30] when the shard is in unassigned state, it is really impossible to tell what nodes are safe to touch unless you look on the other nodes with this shard and make a guess.. https://gitlab.wikimedia.org/repos/search-platform/sre/es-tools/-/blob/main/estools/lib.py?ref_type=heads#L86 [08:44:04] looking [08:50:15] atsukoito: you have to ignore UNASSAGNED shards indeed, but there must be always be a '"primary": true' that's started [08:50:27] or we're already red [08:50:38] oh, good point [08:51:07] if this primary in on the host you plan to remove you have to check that it has at least one replica [08:51:44] but generally you should have no primaries on the host you plan to remove because you banned this nodes, and the primary should have been promoted to another node [08:52:15] in other words, if only replica shards are on the host, it's safe [08:52:28] if there are still primaries it's possibly unsafe [08:52:55] it's sill safe if there's another replica around on a host that's not going to be down [08:53:49] thanks, Im now visualising if the host has yellow shard primary or secondary: `cirrussearch1098 F2 search:dir,omega:dimr* Ss` (one Primary shard, one secondary shard) [08:55:17] I'll add more data in the output to test the heruistic [08:59:20] sure [09:00:11] the thing to keep in mind is that the primary shard is always the source for recoveries, opensearch never copies a replica shard to another replica [09:01:30] it can promote replica to a primary or vice versa quite quickly, this makes the check prone to race conditions [09:03:33] before checking the hosts, they must be banned at the routing level otherwise opensearch might promote some shards to primary there quite quickly (in the short time you run the check and you actually bring down the node) [09:49:39] lunch [11:01:44] what is api to ban/check node for ban? [11:06:27] ah, I see https://www.irccloud.com/pastebin/09Tn4ReN/ [11:50:00] lunch [12:23:11] atsukoito: yes that's it, I believe this step should be part of the runbook [13:06:38] \o [13:06:52] I'm reviving the spicerack automation now that I kinda know this kung-fu, hopefully I'll be able to incorporate it [13:07:14] o/ [13:10:43] o/ [13:13:37] Note that we typically don't ban hosts ahead of time. We typically assume that it's safe to reboot up to 3 hosts if they're on the same row because of the anti-affinity rules we use. That being said, to be truly safe we should ban/drain ahead of time. I did do that once when I was working between rows, it took about 3 hours [13:14:18] i suspect it's reasonably safe to check that every index that is on one of those three nodes also has a replica not on those three nodes [13:14:31] no guarantees due to races, but probably "good enough" [13:15:01] Erik mentioned yesterday that we could tweak `indices.recovery.max_bytes_per_sec` to make recovery go a bit faster, it's something we could experiment with on the smaller clusters [13:15:41] We don't want to too I/O bound though, that will result in cluster quorum issues [13:15:57] i would at least want to ponder and align the max_bytes_per_sec, max recoveries per node, and our data sizes. I feel awkward about commonswiki shards taking 3+ hours to move [13:16:13] Agreed, I'll get a ticket started for that too [13:17:37] inflatador: does banning cause issues? [13:18:14] I mean banning does not necessarily mean waiting for the host to be fully drained [13:18:55] btw: w/ restarts done probably going to start reindexing for redirects today, after double checking that everything is in place [13:19:10] nice! [13:19:26] dcausse it takes a long time, and if you restart the service on a host that is draining it causes shards to get stuck, you have to restart the service on whatever hosts that were receiving the shards too [13:21:27] Adding 3 hours to the beginning of a reimage is annoying, but it's also not the worst thing in the world if we want to avoid losing data [13:21:55] we'd also talked about changing the partitioning recipes so they don't wipe our the OpenSearch data [13:22:52] perhaps it's not necessary to wait for the 3 hours, but wondering if it could help reducing the chance of having primary shards on these hosts [13:24:03] I suspect that the cluster is shuffling shards massively when do these upgrades and I'm just worried that a safety check might possibly be a bit weak if shards are moving around a lot [13:47:34] Definitely a valid point. A lot of the problems can be solved by simply slowing down, and that's not a bad place to be [13:56:27] created T431712 to look at some of the shard recovery options [13:56:29] T431712: Cirrussearch: Test/optimize shard recovery settings - https://phabricator.wikimedia.org/T431712 [14:22:30] Y'all probably saw this already, but posting just in case https://www.danieleteti.it/post/http-query-method-en/ [14:23:15] i'll be excited to use it in 30 years, just like ipv6 [14:23:27] (ok it might not be that bad :P) [14:23:30] LOL [14:23:46] I always wondered why no one ever used SEARCH or PROPFIND, that post explains it pretty well [14:27:16] Damn, it also looks like someone proposed an update to SEARCH that's been around for 5 years https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-00.html [14:27:32] your 30 years comment might be accurate ;P [14:28:27] i do suspect there will be oddities, old corp proxies that don't forward it properly, etc. Maybe 5-10 years is a better guess, but will be interesting to see how it rolls out [14:28:32] maybe corp proxies aren't as much of a thing anymore [14:32:08] API gateways and other proxies would probably have to understand it too [14:42:29] hmm, turns out we now need kubernetes python lib to install cirrus-reindexer, but it doesn't seem to be in system-site-packages [14:42:44] but it should be ... hmm [14:43:12] they changed the deployment host recently, perhaps that's it? [14:43:18] maybe [14:45:40] on deploy2003 I see python3-kubernetes installed [14:46:02] yea curiously i can import it, but pip still wants to reach out to pypi and install it (even though it sees the system provides pyyaml and requests) [14:46:07] maybe versioning of some sort...looking [14:47:01] oh! we pinned kubernetes to 12.0.1, this has 22.6.0 [14:47:05] probably new deploy host [14:47:22] * ebernhardson is separately curious how the major version jumped by 10, seems sus [14:48:05] yea, deploy1003 has 12.0.1, deploy2003 has 22.6.0 [14:48:19] i'm...not yet sure what to do with that :P [14:57:43] reviewing change logs (w/claude)...suggests nothing we use changed [14:57:53] nice [15:12:20] errand [15:23:37] does anyone know where our code to create reindex tickets lives? I might borrow it for T431498 [15:23:38] T431498: Cirrussearch: exercise OpenSearch snapshot restore function on a regular cadence - https://phabricator.wikimedia.org/T431498 [15:24:28] inflatador: hmm, i think it's a couple pieces iirc. There is a prometheus data collector that records how long ago an index was created, then there is a bit in alertmanager that creates an alert, and some existing machinery that makes the alert be a ticket instead of an email [15:26:14] Thanks, that's helpful. I do know how to create tickets w/Alertmanager, so I guess we figure out how to fire every 6 months. Maybe just alert on the age of a test snapshot or something [15:32:27] yea i guess the question is how to model the data so it fits into alertmanager / prometheuses model. I think the method from before where we lookup the creation data and map it to a time-since is reasonable. It's also very silly and a bit wasteful, but thats just life. [15:52:40] * ebernhardson sighs and fixes yet another repo using an image that uses mirrors.wikimedia.org ... [17:09:45] dinner [17:16:38] reindexer started for all three clusters. the first thing it does is commonswiki...so basically start it and wait 12 hours for much to happen :) [18:01:24] LOL [20:51:49] relatively surprised to say that looks like i have opensearch 3.7.0 booted with all our plugins. Will try get cirrus integration tests going next [20:54:59] not sure whats up with cloudelastic alerting :S poking [20:56:22] reindexer claims to have failed twice to reindex commonswiki_file, it failed with some 'node not connected' errors [20:57:12] ahh, the instances were rebooted, that makes sense :) [21:12:38] ebernhardson yep, we were just realizing that too, sorry [21:13:24] will the reindex recover on its own? We got a ticket for reboots today [21:15:10] We can put them off if we need do, but if we already broke something it might be better to get the other 4 cloudelastic hosts out of the way. We can wait on production, eqiad is actually already done [21:15:13] Ref https://phabricator.wikimedia.org/T431658 [21:31:03] Is there a way to check for active reindexes? Maybe in Airflow or K8s api? [21:38:06] looks like the reindex is indeed stuck, based on the lack of index file size changes `curl -s https://cloudelastic.wikimedia.org:9243/_cat/indices/commonswiki_file_1783616177?bytes=b` [21:40:30] I'll delete the empty alias, it started on another one. We'll skip the rest of the reboots and let that finish