[09:26:08] errand+lunch [13:20:32] o/ [13:28:20] I'm having problems with copying the opensearch-madvise reprepro deb pkg. Do we still need a separate program for this or can we use a systemd timer to set readahead a la our k8s setup? [13:32:38] inflatador: no clue, I thought we had setup read_ahead to a lower value for baremetal machines but that does not mean opensearch-madvise is not required [13:32:58] what I read is that from lucene 10 + java 21 this should no longer be required [13:35:31] opensearch 2.x is still on lucene 9.x apparently [13:36:24] inflatador: we can rebuild opensearch-madvise if necessary I think, Erik might know more [13:45:27] dcausse no worries. We are using OpenSearch 3 in k8s and that still needs the readahead? [13:50:25] inflatador: hm.. I think we tuned read aheads in k8s because of it was set to 8mb on top of ceph, if we run os3 + java21 I guess we benefit from the optimization described at https://github.com/elastic/elasticsearch/issues/27748 but suspecting that is not enough with ceph & very high read-aheads [13:54:48] dcausse oh you're right, I was thinking 8mb was the default but I'm wrong about that [13:55:59] maybe we wait till OS 3 to revisit madvise then [13:56:41] sure, but let's wait for Erik, I'm not too familiar with all these optimizations [14:19:21] np [14:48:15] speaking of packaging, looks like upstream 2.x owns `/usr/share/opensearch/plugins/opensearch-knn/LICENSE.txt` but our `wmf-opensearch-search-plugins` is trying to write there as well. Any preferences on how to deal with this? Here's the list of knn-related files provided by upstream pkg: https://phabricator.wikimedia.org/P90675 [14:50:34] Looks like the 2.x package already provides the `opensearch-knn` stuff, guessing we can just remove it from our pkg [15:06:39] ryankemper we're in standup if you wanna join [16:05:05] inflatador: re /usr/share/opensearch/plugins/opensearch-knn/ could this mean that the deb already include this plugin and we try to install it twice? [16:06:03] yes I think upstream deb is similar to their docker image and includes a bunch of plugins :/ [16:06:05] dcausse yeah, I think I'm the one that added that but I can't remember why. Maybe for Semantic Search? [16:06:17] ah perhaps? [16:06:39] but we never used opensearch 2 for semanticsearch [16:07:20] True. Maybe during the early testing process before y'all switched to 3x? [16:07:35] no clue :/ [16:08:02] I think Erik got the cirrus devel image working, looking at what baseimage it's using [16:10:50] it's using opensearchproject/opensearch:2.19.5 (https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/blob/main/.pipeline/blubber.yaml?ref_type=heads) [16:11:03] possible that we might just override things here [16:11:40] ah no we just drop all upstream plugins at https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/blob/main/.pipeline/blubber.yaml?ref_type=heads#L37 [16:12:16] inflatador: so if possible a deb with no plugins at all would be great [16:16:02] dcausse I'm trying to get away from building a custom opensearch deb, can you help me understand the risks of using their plugins? Like could they clobber some of ours or something? [16:19:59] inflatador: yes but there are plugins we really don't want on the prod cluster, like the top-query thing is too risky to install imo [16:20:48] oh yeah, the performance analyzer? [16:21:05] yes... [16:21:15] Let me add that to our requirements, that thing def sucks ;( [16:21:32] and possibly other things we haven't looked into, there are so many plugins installed by default [16:22:46] I'm fine with being careful about plugins, I'm just not sure if we want to make a custom deb pkg. (Although I wouldn't rule it out either, their debs are not good) [16:25:58] sure let me know if we can help on this [16:31:05] Cool, I've updated T423291 to add disabling unwanted plugins to be part of the requirements. I think I would rather do it with puppet than messing with the upstream pkg but I could be convinced otherwise [16:31:05] T423291: Build new wmf-opensearch-search-plugins package for opensearch 2.x/trixie and ensure we don't install/enable any unwanted plugins in prod - https://phabricator.wikimedia.org/T423291 [16:37:54] the upstream deb is bloated with a bunch of things... [16:38:15] a fully-fledged jdk and 26 plugins :( [16:38:47] what do we do for opensearch 1? [16:40:16] Observability built their own pkg before upstream had debs, that's what we use for 1.x [16:42:25] ah got it, now they use the upstream opensearch2 deb? [16:44:25] No, they also built their own opensearch 2.x deb and now they are stuck on 27 which is ~3y old. [16:45:02] why are they stuck? [16:45:19] Some cross-dependencies with logstash I think? [16:45:46] They tried to update to 2.19 when I first mirrored the upstream repo and it broke some of their stuff [16:46:34] T410795 [16:46:36] T410795: Logstash: No longer able to install OpenSearch 2.7.0 from internal apt repo - https://phabricator.wikimedia.org/T410795 [16:46:50] they're also maintaining their own version of curator [16:48:24] curator has not been ported to opensearch I guess? [16:48:36] pretty much a pain I agree... [16:49:22] yeah, it's pain no matter what approach ;P. opensearch has `opensearch-dashboards index lifecycle management` now which is basically curator [16:49:57] if we go the upstream approach we need to carefully check what plugins are getting in and having a way to disable them via opensearch.yaml [16:50:19] and then dedup what we have in the wmf-plugins [16:51:01] but not sure how to truly disable installed plugins... [16:51:28] Agreed, I'll add that to the requirements too. I'm hoping we can get enough leverage with upstream to get them to fix things with their deb. We've gotten them to move on a few things [16:52:28] sigh.. there's even a opensearch-skills now installed by default [16:54:01] https://github.com/opensearch-project/opensearch-build/pull/5554 I managed to get some attention on this yesterday. Maybe I can propose an `opensearch-minimal` pkg that doesn't have any plugins [17:01:07] would be great, there's a tar distrib with opensearch-min-3.6.0-linux-arm64.tar.gz at https://opensearch.org/artifacts/by-version/ if they could make a deb that'd be awesome [17:01:26] ooh, nice find! Thanks [17:02:25] dinner [20:45:20] Created T423327 to talk more about the opensearch packaging situation, feel free to respond there [20:45:20] T423327: Explore options for OpenSearch 2.x/3.x packaging and distribution - https://phabricator.wikimedia.org/T423327