[10:33:56] <dcausse>	 just discussed with joseph and looked at the data, only ~100k, 25k and 5k pages are changed on enwiki, frwiki and ptwiki respectively over one week, I'm changing strategy and will index the diff
[10:43:48] <pfischer>	 o/ dcausse, regarding https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1243696: According to Guillaume, they cannot accept the patch because that would result in >70% utilisation of the k8s cluster, which outside the comfort zone. SRE wanted to get back to us with affordable numbers.
[10:44:58] <dcausse>	 pfischer: this patch is now just about testing frwiki to get a sense of the IO and minimal RAM requiremens
[10:47:40] <dcausse>	 we initially request 768Gb of ram which based on https://grafana.wikimedia.org/d/WG4NjDISk/cluster-status-and-capacity?orgId=1&refresh=1m&var-datasource=000000026&var-site=eqiad&var-cluster=k8s-dse&from=now-30d&to=now&timezone=utc&viewPanel=panel-2
[10:48:03] <pfischer>	 Oh, I missed Balthazar’s comment. Alright. So this would definitely only fit frwiki at max?
[10:48:13] <dcausse>	 peak usage seems around 1.2Tb
[10:49:10] <dcausse>	 1.2Tb + 768 is ~2Tb which I think is still under 70%
[10:49:52] <dcausse>	 pfischer: this frwiki test is just for us to learn how opensearch behaves with remote volumes and how low we can in term of ram
[10:50:55] <dcausse>	 but I'm open to see what we can do with the max they can give
[10:51:55] <pfischer>	 Understood. Thanks! I’ll get back to SRE and unblock this either way.
[10:52:18] <dcausse>	 thanks!
[10:56:12] <dcausse>	 lunch
[14:09:09] <inflatador>	 <o/
[14:32:08] <ebernhardson>	 \o
[14:32:33] <dcausse>	 o/
[14:38:38] <inflatador>	 .o/
[15:18:49] <inflatador>	 Trey314159 sorry for the late scratch! I have to get ready for a capacity planning mtg at the top of the hour. Hooray
[15:19:29] <Trey314159>	 no worries!
[15:47:20] <dcausse>	 I feel that writing rows like '{"_index":123}\n{...}' as text is going to hit me at some point... wondering what to do, replace \n with a kind of separator or not writing as text...
[15:48:00] <ebernhardson>	 dcausse: hmm, i don't follow the concern?  Is it that we might miss the \n?
[15:48:49] <dcausse>	 it's that it'll end up being two lines in the dataset and if for some it's re-shuffled it might break the bulk format? 
[15:49:07] <dcausse>	 s/some/some reasons/
[15:49:34] <dcausse>	 spark.read.text("/user/dcausse/semantic_search/ptwiki_full_paragraphs_20260215.ndjson").take(1) -> [Row(value='{"index": {"_id": 309}}')]
[15:50:00] <dcausse>	 but also at read time, I need to read two lines then?
[15:50:01] <ebernhardson>	 ohh, yes if it shuffles or treats things as per-line that could totally break things....hmm
[15:51:25] <ebernhardson>	 i guess in convert_to_esbulk i solved that by having a very short path, basically repartition->format to pairs->saveAsTextFile
[15:51:38] <ebernhardson>	 keeping it in row format until it's ready to write out
[15:53:23] <ebernhardson>	 reading is more tricky...i think i delt with that somewhere but forget where...hmm
[15:53:56] <ebernhardson>	 naively, i suppose i would read->mapPartitions to pair it up, but not sure where i've done that before
[15:54:04] <dcausse>	 also I'll have deletes which are single lines...
[15:54:28] <ebernhardson>	 :(
[15:54:41] <dcausse>	 perhaps I keep a structured format with two columns and concat at the very end?
[15:55:08] <ebernhardson>	 seems reasonable, one way or another it probably mostly needs to be processed with the unit of work as a single item
[15:55:19] <ebernhardson>	 only spreading out/in at the very edges
[15:55:49] <dcausse>	 or everything wrapped as a json: {"header": {"index": ...}, "data": {}) that I re-interpret later
[15:57:12] <dcausse>	 but I liked the idea of being able to do hdfs dfs -text /part | split/parallel -> curl
[16:00:17] <ebernhardson>	 hmm, yea it is convenient to use parallel and curl, needing a pyspark script to ship the changes is less than ideal.  But maybe we could have a generic data shipper?  A generic shape and a generic script we can reuse?
[16:00:33] <ebernhardson>	 similar in idea to the mjolnir bulk daemon, but simpler and more specialized?
[16:00:39] <ebernhardson>	 (and not a daemon)
[16:01:47] <pfischer>	 dcausse: are you around for the resource sizing meeting?
[16:01:53] <dcausse>	 oops
[16:42:37] <ebernhardson>	 dcausse: are we going to use index aliases?  And should cirrus expect some alternate index suffix, frwiki_semantic or some such? Or should we just call it frwiki_content?
[16:48:03] <ebernhardson>	 calling it frwiki_content is probably the easiest
[17:09:46] <dcausse>	 yes I was planning to add alias and keep _content for this one
[17:10:49] <ebernhardson>	 i was also pondering where we set the search_pipeline, it seems like the qwen3-embedding could simply be the default pipeline set in index settings, it only defines the default model to use for neural searches to embed their queries
[17:11:16] <dcausse>	 sure
[17:11:18] <ebernhardson>	 but longer term if we introduce reranking it's not clear if that should be default or not.  I suppose as long as it's a dedicated cluster it can be, but otherwise need to thread through to add the query param in cirrus (probably not that hard)
[17:12:18] <dcausse>	 yes... all these search_pipeline bits are still very fuzzy to me...
[17:16:09] <ebernhardson>	 i'll try and wrap up the cirrus bits today, if they want a plausible api end of next week it needs to be merged before mid-day eu on monday
[17:16:24] <dcausse>	 ack
[17:16:43] <ebernhardson>	 skipping search pipeline in cirrus for now, assuming we set the qwen3-embedding as default
[17:17:08] <dcausse>	 can this be set after the fact on the index?
[17:17:08] <ebernhardson>	 well, i might spend an hour to see where it fits in, if easy enough will attach it to the Search objects
[17:17:43] <ebernhardson>	 yes, it's an index setting, `index.search.default_pipeline`
[17:17:56] <dcausse>	 this means all searches will go through this pipeline?
[17:18:13] <ebernhardson>	 yes, but in the case of qwen3-embedding it doesn't actually do anything, it just sets a default model that a neural query looks up
[17:18:24] <dcausse>	 ok
[17:18:26] <ebernhardson>	 the reranking one does though, and probably doesn't make sense as a default
[17:19:10] <ebernhardson>	 i suppose the main thing the qwen3-embedding is doing is avoiding needing arbitrary model ids (that change per cluster) in mw config
[17:25:26] <dcausse>	 yes that model id was going to be a pain without this search pipeline thing
[17:25:54] <dcausse>	 watching the cluster doing a rolling after helmfile apply :)
[17:26:46] <ebernhardson>	 i imagine for now, you are manually creating the index settings/mapping?
[17:26:53] <dcausse>	 yes...
[17:27:23] <ebernhardson>	 i can maybe fit that into the cirrus-toolbox bits, index templates are another stateful api
[17:27:29] <ebernhardson>	 (but later, not this week)
[17:27:47] <dcausse>	 sure
[17:27:58] <dcausse>	 hm... it did not pick up my jvm settings
[17:28:18] <dcausse>	 still seeing -Xms1g, -Xmx1g,
[17:28:21] <ebernhardson>	 :S
[17:28:47] <ebernhardson>	 should be able to kubectl exec into the host via the -deploy credentials and see what actually got passed
[17:29:18] <inflatador>	 you might have to run `env` in the pod itself, it doesn't update the configmaps
[17:29:27] <dcausse>	 it just added new extra pods but did not seem to pick my new mem setting
[17:29:44] <inflatador>	 oh yeah, that's a known issue. The operator basically does nothing besides stand up the cluster and replace dead pods
[17:30:21] <inflatador>	 https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/OpenSearch-on-K8s/Administration#Changing_Resources_on_a_Live_Cluster 
[17:30:37] <inflatador>	 it's annoying, but deleting the pods is really fast b/c it doesn't actually delete the data
[17:30:59] <dcausse>	 can I delete the pods myself?
[17:31:34] <inflatador>	 yeah, you need to do `kube_env opensearch-semantic-search-deploy dse-k8s-eqiad` , the regular user doesn't have perms
[17:31:36] * ebernhardson loves pods: OCI runtime exec failed: exec failed: unable to start container process: exec: "ps": executable file not found in $PATH: unknown
[17:31:55] <ebernhardson>	 sure i could dig through /proc, but thats not fun :P
[17:32:08] <inflatador>	 I'd say use nsenter but ya gotta be root for that ;(
[17:32:43] <dcausse>	 hm.. I see duplicated Xmx1G and then Xmx4g on the new pods...
[17:35:22] <ebernhardson>	 yea same, -Xmx4096M comes last and should win...
[17:35:38] <dcausse>	 ok
[17:36:48] <inflatador>	 Where are y'all setting it? I can take a look at the chart
[17:37:05] <ebernhardson>	 i hadn't realized it would munge the arguments,  /proc/pid/1/cmdline shows -Xmx4096M, but the opensearch startup says -Xmx4G
[17:37:21] <ebernhardson>	 it's same, i just didn't expect it to normalize it
[17:37:28] <dcausse>	 I need to find the settings to kill these audit & top-queries indices and the like, not super keen having shards being created at random times
[17:37:36] <ebernhardson>	 same
[17:37:49] <inflatador>	 Yeah, that's something I'd like to apply via whatever default cluster settings thingie we come up with
[17:38:35] <ebernhardson>	 inflatador: oh i should mention, i went with a narrow cluster settings bit in the cirrus-toolbox that's managing the other stateful bits. It's currently limited to only effecting plugins.ml_commons.* parts of /_cluster/settings
[17:38:53] <ebernhardson>	 since there is a direct dependency between the config (it references hosts) and the settings (it has a host whitelist)
[17:39:47] <ebernhardson>	 dcausse: i guess make a ticket? I can probably look into which plugins to remove next week
[17:40:25] <ebernhardson>	 looks like it did pick up the 4g now: [opensearch-semantic-search-masters-0] heap size [4gb], compressed ordinary object pointers [true]
[17:40:44] <dcausse>	 yes just restart the first original pods
[17:40:47] <ebernhardson>	 ahh ok
[17:40:48] <dcausse>	 *ed
[17:41:04] <dcausse>	 I guess I can create the index now
[17:42:52] <ebernhardson>	 apparently there are cluster settings to disable the top_queries indices: https://docs.opensearch.org/latest/observing-your-data/query-insights/settings-api/
[17:42:54] <ebernhardson>	 lemme see
[17:49:07] <ebernhardson>	 applied cluster settings to both that should (probably?) disable the top_queries indices from being created and deleted the existing top_queries indices
[17:49:37] <ebernhardson>	 the security auditlog will probably still be created, i dunno what we want to do with that. Doesn't seem like the worst idea, but we would need a cleanup routine
[17:49:49] <dcausse>	 ok it's pushing data to frwiki_content_20260215
[17:50:29] <ebernhardson>	 or maybe it clears old audit indices on it's own, not clear. It looks like the query-insights plugin might have an auto-cleanup at least
[17:52:27] <ebernhardson>	 oh hmm, looking at the query i generate in the semantic search also need to disable suggester, shouldn't be too hard
[17:54:43] <dcausse>	 seems very slow :/
[17:54:48] <dcausse>	 comparing to relforge
[17:55:17] <ebernhardson>	 index is still red, can't be helping
[17:55:55] <dcausse>	 oh
[17:55:58] <ebernhardson>	 6 nodes and 7 shards
[17:56:13] <dcausse>	 did I messed something?
[17:56:37] <dcausse>	 one node died?
[17:57:08] <ebernhardson>	 hmm, i see 7 in deployment-charts.  Dieing would be suspicious, we have nodes 0,1,2,3,4,5. Maybe 6 died, but seems like maybe it never got started?
[17:57:37] <ebernhardson>	 random possibility, denied for resource limits?
[17:57:37] <dcausse>	 java.lang.UnsatisfiedLinkError: no opensearchknn_faiss_avx512 in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
[17:58:02] <ebernhardson>	 oh indeed, there was a 6, logs are avaliable
[17:58:27] <dcausse>	 I use kctl logs pod_id --previous on the node that restarted 
[17:58:33] <ebernhardson>	 hmm, but then how do the others work :S
[17:58:56] <dcausse>	 this is a setting we had to add IIRC on relforge initially
[18:00:06] <dcausse>	 all failed
[18:00:08] <dcausse>	 :)
[18:00:08] <ebernhardson>	 fwiw, that lib is in plugins/opensearch-knn/lib
[18:00:14] <dcausse>	 just ran _count on the index
[18:00:39] <ebernhardson>	 i guess it's called libopensearchknn_faiss_avx512.so
[18:00:44] <dcausse>	 where should this go? the image or helm values?
[18:01:19] <inflatador>	 Interesting, I wonder we need some kind of AVX device passthrough?
[18:01:20] <ebernhardson>	 can it be set from helm? Seems more changable there
[18:01:40] <ebernhardson>	 baking things into the image when we aren't 100% sure just seems time consuming i guess
[18:02:14] <dcausse>	 -Djava.library.path=$OPENSEARCH_HOME/plugins/opensearch-knn/lib should work in helm values
[18:02:37] <dcausse>	 but not sure about "$OPENSEARCH_HOME" might need to hardcode the path
[18:02:55] <ebernhardson>	 yea i'm not sure the context that runs in either
[18:04:07] <inflatador>	 you should be able to exec into the pod and run env, that might help nail down the OPENSEARCH_HOME value
[18:04:20] <ebernhardson>	 it should be /usr/share/opensearch
[18:04:25] <ebernhardson>	 (from `pwd`)
[18:04:27] <dcausse>	 ok
[18:04:31] <inflatador>	 yeah, that sounds right
[18:06:04] <dcausse>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1244729
[18:06:10] <inflatador>	 lunch, back in ~1h
[18:12:50] <dcausse>	 I deleted all pods at the same time perhaps not the best idea, I wonder if the operator is waiting for a sane state before going to the next pod
[18:13:03] <dcausse>	 seems kind of stuck :/
[18:13:35] <ebernhardson>	 :S
[18:13:45] <ebernhardson>	 delete the cluster? In theory that's what this should be good at 
[18:13:51] <ebernhardson>	 can re-apply settings
[18:14:32] <dcausse>	 you mean helmfile undeploy?
[18:14:45] <ebernhardson>	 yea, basically remove it from helm and re-initialize
[18:14:55] <ebernhardson>	 would lose all the data, but nothing important there
[18:15:03] <ebernhardson>	 well, actually i wonder what happens in ceph
[18:15:20] <ebernhardson>	 equally plausible it just gets abandonded and lives forever? 
[18:15:39] <ebernhardson>	 or it gets assigned the same namespace/paths
[18:15:40] <dcausse>	 yes no clue :/
[18:15:53] <ebernhardson>	 me either, fun with new infra :)
[18:15:55] <dcausse>	 the initial deployment mentions a cluster-bootstrap-0
[18:16:59] <dcausse>	 ok time to learn then
[18:17:18] <dcausse>	 all gone
[18:17:25] <ebernhardson>	 :)
[18:19:27] <ebernhardson>	 dcausse: btw for fields, i think cirrus is going to want to see namespace/title/namespace_text/wiki/timestamp. 
[18:19:52] <ebernhardson>	 surprisingly, os doesn't seem to complain about unknowns in `_source` or `fields`, but cirrus might be surprised later
[18:20:28] <ebernhardson>	 we can always stuff those in later though, title is there and the namespace/namespace_text/wiki should be constants across the whole index since it's only NS_MAIN
[18:20:50] <dcausse>	 ebernhardson: this is what I have: https://phabricator.wikimedia.org/P89064
[18:21:24] <ebernhardson>	 oh yea that should be plenty. Thanks!
[18:23:02] <dcausse>	 sigh it seems in a kind of error loop join validation on cluster state with a different cluster uuid zgn4ztgOTnGdW3g6NImERg than local cluster uuid zr2lEu_QQ3iB_xF6SlRUqg, rejecting
[18:23:09] <dcausse>	 seems like data on disk was kept
[18:23:57] <dcausse>	 not sure what to do... wipe-out manually some state within the pod to help it?
[18:24:13] <dcausse>	 or recreate the ceph volumes?
[18:24:39] <dcausse>	 I guess I broke it for real and I now need an adult :P
[18:24:48] <ebernhardson>	 hmm, yea deleting /var/lib/opensearch/nodes/0 might work
[18:24:58] <ebernhardson>	 lol
[18:25:21] <ebernhardson>	 i would be tempted to delete the ceph volumes somehow, but not sure if it initializes that itself or if it was done separately
[18:25:32] <ebernhardson>	 I'm going to guess it creates them itself, it would have to to add nodes?
[18:26:28] <dcausse>	 no clue :/
[18:27:24] <dcausse>	 well... Node 'opensearch-semantic-search-masters-0' initialized
[18:27:37] <dcausse>	 -1 is booting
[18:27:45] <dcausse>	 but will probably fail the same way
[18:29:36] <dcausse>	 ok doing rm /var/lib/opensearch/nodes/0 on every new nodes
[18:42:23] <dcausse>	 ok seems back
[18:42:47] <dcausse>	 ebernhardson: I guess I broke your settings as well, is it easy to reship them?
[18:43:40] <dcausse>	 hm not sure... it did not complained when I re-created the index with the search_pipeline
[18:43:45] <ebernhardson>	 yea super easy, sec
[18:44:14] <ebernhardson>	 well, if the cluster is happy :P  {"error":{"root_cause":[{"type":"m_l_exception","reason":"Fetching master key timed out."}],"type":"m_l_exception","reason":"Fetching master key timed out."},"status":500}
[18:44:32] <ebernhardson>	 hmm
[18:45:05] <dcausse>	 :/
[18:45:07] <ebernhardson>	 oh there it goes, just needed a sec
[18:45:13] <ebernhardson>	 should all be applied now
[18:45:17] <dcausse>	 thanks
[18:45:29] <dcausse>	 indexing now
[18:45:53] <dcausse>	 dinner, will check back a bit later
[18:46:38] <ebernhardson>	 hmm, i guess i need to figure out a variant of that config that can be applied to a local cluster...maybe with llama to provide the embedding
[18:51:06] <ebernhardson>	 hmm, do we pass the fulltext profile into SemanticResultsType, or add some sort of SearchContext::$resultsTypeOptions to allow the query builder to provide the options
[18:51:26] * ebernhardson realized he has hardcoded options in SemanticResultsType that are parameters to the SemanticQueryBuilder
[18:52:10] <ebernhardson>	 i guess i feel better not adding even more indirection...seems plausible to pass them in directly
[19:03:27] <dcausse>	 sigh... java.lang.UnsatisfiedLinkError: /usr/share/opensearch/plugins/opensearch-knn/lib/libopensearchknn_faiss_avx512_spr.so: libopensearchknn_util.so: cannot open shared object file: No such file or directory
[19:07:53] <dcausse>	 it needs LD_LIBRARY_PATH as well I guess
[19:14:10] <dcausse>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1244764
[19:15:31] <ebernhardson>	 LGTM, +2
[19:15:58] <inflatador>	 back
[19:18:04] <dcausse>	 it's slowly crashing all pods... hoping that it won't be stuck again :/
[19:19:45] <dcausse>	 applied will wait a bit to see if it can recover
[19:21:02] <dcausse>	 phew... yes it recovers
[19:22:13] <dcausse>	 spoke too soon
[19:26:12] <dcausse>	 yes.. when it's in this state it does not want to pickup changes and will refuse to spin up deleted pods
[19:26:59] <dcausse>	 deleted opensearch-semantic-search-masters-6 and it's not willing to re-adding it while the cluster is in this bad shape
[19:27:14] <dcausse>	 well... giving up for today
[19:31:27] <inflatador>	 Did you completely wipe out the cluster and try to bring it back? If you do that without deleting the persistent volume claims it won't form a new cluster. I'll add that to the docs
[19:33:48] <ebernhardson>	 inflatador: we weren't sure how to drop the ceph stuff, so instead issued a manual rm -rf on the nodes/0 directory in ceph
[19:33:58] <ebernhardson>	 (not sure if he did this time around, but it worked the first time to reset the cluster)
[19:34:19] <ebernhardson>	 well, the rm -rf was from the host that had it mounted, but the effect was to clear out the node state in ceph
[19:34:45] <ebernhardson>	 s/host/container/
[19:37:45] <inflatador>	 ebernhardson, ACK. I'll add how to do it to the docs, after that I can wipe everything out if you'd like
[19:40:33] <ebernhardson>	 inflatador: sure, docs on how to re-create would be awesome, if you have time to also run it and verify would be great
[19:41:00] <ebernhardson>	 on the upside, it turns out i need all this extra config for my local test instance too...shouldn't be surprised
[19:58:44] * ebernhardson hearts java error messages. Never enough context:  Invalid URI. Please check if the endpoint is valid from connector.
[19:58:50] <ebernhardson>	 like, you could at least print the URI you tried to use :P
[20:01:05] <inflatador>	 LOL
[20:01:41] <inflatador>	 Just redeployed the cluster. Docs on how to completely wipe it here: https://w.wiki/H$ru
[20:06:17] <ebernhardson>	 thanks!!
[20:11:02] * ebernhardson apparently never remembers that _ is not a valid hostname character
[20:11:57] <ebernhardson>	 blame arpanet apparently :P
[20:18:35] <ebernhardson>	 i suppose a side benefit of knn...the results don't have to be any kind of related. Stuff any 5 docs into the index and they will be returned by knn
[20:19:16] <ebernhardson>	 (maybe a downside too, better not deploy knn to tiny wikis :P)
[20:21:16] <ebernhardson>	 randomly curious, cluster has been deployed 20 minutes, first attempt to upload an ml connector still failed with the timeout bit. Re-issuing request 2s later worked fine.  weird
[21:11:37] <dcausse>	 restarted frwiki import, working fine finally :)
[21:12:08] <ebernhardson>	 thanks!
[22:08:08] <ebernhardson>	 hmm, wmf-config really wants a labs service