[07:42:56] yes seems like a regression with opensearch2, wildcard expansion with top_terms_1024 is now hitting the max bool clauses unsure why... [07:57:58] unlike before it seems that the limit is applied globally somehow [08:04:15] sigh... "persistent setting [indices.query.bool.max_clause_count], not dynamically updateable"... [08:04:18] https://issues.apache.org/jira/browse/LUCENE-8811 [08:04:35] going to be fun... [08:35:33] hiya! [08:35:44] dcausse: do you want me to depool codfw? [08:35:51] hi! [08:37:27] atsukoito: I was about to ask, I not clear if this deserves a depool or not... graphs show that 0.4requests/sec are failing because of this, this is not huge but not nothing... [08:37:53] filed T431086 [08:37:53] T431086: Searching with wildcards fail with OpenSearch 2 - https://phabricator.wikimedia.org/T431086 [08:38:47] yes probably a depool is preferable... it's going to cause much confusion esp. because only codfw users are affected [08:39:52] we can ask gehel and pfischer if they have opinion on this ^ [08:40:37] i think it worth depooling while we investigating options as well. [08:41:26] yes you're right, I don't see an easy workaround at this point... and it's going to take a while before we come up with a sane solution (it's not going to be fixed today for sure) [08:48:20] atsukoito: ok, let's depool if you're OK with this, I'll respond to the village pump once done [08:50:05] I don't have a strong opinion. Depools are cheap, so let's do it and see if we can find a fix [08:51:19] gehel: thanks [08:51:51] atsukoito: confirmed the depool fixed the issue [08:51:53] thanks! [08:52:04] i've issued depool, waiting for 5 minutes to fully propagate [08:55:09] dcausse: thanks for confirming, I've reassigned T431091 to you for tracking, feel free to do anything with it :) [08:55:09] T431091: Regression in search with OpenSearch 2 - https://phabricator.wikimedia.org/T431091 [08:55:21] thanks! [08:56:38] actually, sorry for the duplicate, I'll merge it [08:57:51] np! [09:10:58] SRE/Product question: do we have an application-level log somewhere in logstash/events/metrics that shows aggregate query stats? [09:12:13] E.g. search request of this metadata from this wiki at this dc, served by this cirrussearch, failed/succeded with this time [09:13:18] Would be nice to have, and also have the metadata like number of terms so we can adjust `top_terms`. [09:23:18] we do have bunch of events in kafka, see https://stream-internal.wikimedia.org (shameless plug), that have search in its name, looking now [09:29:07] we track global search failures at https://grafana-rw.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1&from=now-7d&to=now&timezone=utc&var-cluster=elasticsearch&var-exported_cluster=production-search&viewPanel=panel-9 [09:29:40] here we see the bump in dnsdisc.unknown [09:30:39] I'm not clear why dnsdisc.rejected increased as well... [09:31:36] finding the right settings is going to be tough tho... decreasing top_terms might have user impact in such a way that recall will be affected [09:32:48] I'll have to check with the team but seems like the way forward is possibly to bump the cluster wide limit to something that's rarely hit and add a more user-friendly message when the limit is hit [09:33:52] but at this point I don't how we could go back to the previous opensearch 1.x behavior (unless we bump the cluster wide limit to some insane value like 2³¹ [09:35:39] as for investigating the number of boolean clauses a query is going to create, it's quite hard to do that just looking at logs, you need terms stastitics from the search index, in other words you have to replay the query [10:03:12] thanks! [10:21:16] lunch [13:06:20] sigh was checking the wrong cluster earlier today, indices.query.bool.max_clause_count is actually updatable in codfw [13:28:43] so now the question where do we put indices.query.bool.max_clause_count (opensearch.yml, one-off curl to cluster settings, cluster settings update via existing maint scripts)? [13:28:50] and to what value... [14:23:25] dcausse: what's `opensearch.yml` [14:25:30] atsukoito: the opensearch config file, it's a template at https://gerrit.wikimedia.org/g/operations/puppet/+/9d6aaeba2a7efb53f848368820fb102fc2eb70dd/modules/opensearch/templates/opensearch_2.yml.erb [14:25:37] ah, thanks! [14:26:14] we do add some settings there, still unclear if indices.query.bool.max_clause_count would be a good fit [14:26:26] caveat with this file is that it requires a restart [14:27:06] i would say both `opensearch.yml` (in case we'd need to wipe cluster one day) and `curl` for live application now [14:27:19] atsukoito: sounds good [14:27:42] do you need any assistance with it? [14:28:07] running some analysis to determine what would be a good threshold, yes after that we'll need a puppet change and run some _cluster/settings updates [14:28:40] cool, I'll be around [14:54:23] ok will go with 5 wildcards (conservative), 5*1024*2=10240 should I think fail very few queries [14:56:02] dcausse: do you want to make a puppet change yourself or should I make it? [14:58:26] https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/search-settings this setting is deprecated? [14:59:32] oh, only deprecated in ES, but exists in OS2, https://docs.opensearch.org/latest/install-and-configure/configuring-opensearch/index-settings/ [15:12:58] making a diff [15:14:25] atsukoito: sorry had to step out for a bit, I can push a change unless you have something almost ready [15:14:54] you mean puppet change? i already written it, writing a commit message [15:15:20] yes, thanks for doing it! [15:15:28] updating cluster settings now [15:18:22] dropping archived.search.remote.*.seeds settings from 9643 [15:18:44] https://gerrit.wikimedia.org/r/c/1307470 will test with pcc [15:21:06] ok settings updated [15:22:36] but we perhaps want to wait for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1307372 to be shipped before actually bringing codfw online [15:23:08] that means running eqiad only for the week-end, I suspect it's fine but please let me know if you prefer otherwise [15:28:27] sigh... search query logs contain more PIIs than ever with those AI assistants hitting our search endpoint.... [15:31:45] i think i'd like to try pooling back the cluster as we changed the parameters. leaving only single cluster over the weekend is kind of scary [15:32:30] atsukoito: sure [15:34:00] https://puppet-compiler.wmflabs.org/output/1307470/7134/cirrussearch2073.codfw.wmnet/index.html [15:34:14] btw, did you push the setting for all three clusters? [15:34:53] atsukoito: yes [15:35:05] testing a query on the two small ones [15:37:18] atsukoito: all good you can repool anytime [15:38:16] repooled back [15:39:22] lemme try testing it from mw-debug [15:40:04] still hitting eqiad for now [15:40:26] it just flipped on some requests [15:42:08] ok seems to work, getting the failures when using 5 pathological wildcard queries (a* b* c* d* e*) [15:42:17] fine otherwise [15:44:34] atsukoito: thanks for the help on this! [15:49:51] the "Cirrus failures" graph remains almost flat near 0 for "dnsdisc.unknown" that's a good sign, logstash reports only my own test queries [15:52:42] cool! could you please share where do you look for the logstash errors? [15:53:02] i updated the Village pump, too. [15:53:55] atsukoito: https://grafana-rw.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1&from=now-3h&to=now&timezone=utc&var-cluster=elasticsearch&var-exported_cluster=production-search&viewPanel=panel-9 [15:53:57] thanks! [15:54:38] it's dnsdisc.unknown, the small bump is me testing [16:06:34] heading out, have a nice week-end!