[07:37:47] o/ [08:14:21] o/ [08:55:22] mjolnir completed its 2025-01-17 run [08:56:33] 2025-02-06 has begun. I'll wait to see how it goes before attempting any further memory or configuration tweaks. [09:00:00] \o/ [09:22:25] the rate ext.pageassessment.project weighted tags is down to a dozen per minute with a followup on the optimization shipped yesterday [09:23:10] will resume the backfill of articlecountry tags which I paused yesterday due to high lag [09:23:18] \o/ [09:24:11] dcausse question re tags weights. I added an atriclecountry tag locally with a weight of 1000 [09:24:23] but it got indexed with a weight of 1 [09:24:49] is it expected? [09:24:55] gmodena: you mean in cirrusDump you see "|1" [09:24:59] yes [09:25:07] updated with: docker compose exec mediawiki php maintenance/run.php ./extensions/CirrusSearch/maintenance/UpdateWeightedTags.php --page Plant --tagName "Turkey" --tagType "classification.prediction.articlecountry" --weight 1000 [09:25:34] and cirrusDump reports [09:25:35] "classification.prediction.articlecountry/Turkey|1" [09:25:45] gmodena: should be "|1000" so I guess yet another with the UpdateWeightedTags maint script? [09:26:09] we never use this maint script so it's very possible that it's completely broken... [09:26:16] thought so, but wanted to validate :) [09:26:19] I'll have a look [09:29:36] mmm actually, yesterday we did remove a cast to float that might have caused this normalization (val / 1000). Maybe I messed up while switching gerrit patches [09:36:10] ah! https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1120583 [09:36:22] CirrusSearch patches don't auto-submit on +2 ? [09:46:54] gmodena: oh, probably the -1 from cindy blocked the merge looking [09:47:55] removed an extra newline [09:47:56] dcausse no rush! [09:49:24] ack [09:50:28] ah, I thought that the last +1 cindy gave on patchset 1 (after you removed its vote) meant the CR was ok. [09:50:58] how did you spot the blocking issue? [09:52:49] gmodena: it's just a supposition but the -1 arrived right before the gate pipeline succeeding and I suppose that there's a double check that no -1 is present on the patchset before merging [09:53:20] got it [09:53:46] I thought that only a -2 could shortcircuit a pending merge but possibly a -1 can too [10:13:48] dcausse ack [10:30:19] hm... cindy is not having a good day... [10:31:47] :( [10:32:12] meh... "failed to remove network mwcli-mwdd-default_dps: Error response from daemon: error while removing network: network mwcli-mwdd-default_dps id a0be02477e8881527c68bc960bf038c790498af5b0af9dc3aaad37df2ca683f9 has active, panic: exit status 1" [10:32:47] is it a cindy issue, or flaky ci containers? [10:35:49] dcausse i think i have the basic articlecountry functionality implemented. Possibly it's borked in ways I did not figure out :). I'll open a WIP after lunch, ok? [10:35:59] it's more a problem with docker & mwcli but that's very obscure to me [10:36:15] gmodena: sounds good! [10:49:47] I think docker is borked... [10:50:01] docker network inspect mwcli-mwdd-default_dps shows no containers... [10:52:29] going to reboot the os :/ [10:54:46] well restarting the docker service did the trick... [11:08:08] oof [11:28:38] lunch [14:10:27] o/ [14:13:34] o/ [14:13:50] small break+sunlight exposure. Back in 20ish [15:18:09] gmodena: I've moved your doc to the Search Platform shared folder [15:21:06] I also added a few links [15:31:15] gehel thanks! [15:52:32] I'll skip the Wednesday meeting, conflicting experimentation platform meeting [17:00:23] workout, back in ~40 [18:16:04] looks like there's a typo in the opensearch plugin mapping here https://gerrit.wikimedia.org/g/mediawiki/extensions/CirrusSearch/+/7181ee7ead323f90b6e932e8c3905922761fd650/includes/Maintenance/Plugins.php#9 [18:16:33] `opensearch-analisys-stconvert` should be `opensearch-analysis-stconvert` . The plugin was built with the typo too [18:56:22] created T386860 to talk about the CPU performance governor changes we discussed earlier [18:56:23] T386860: Enable CPU performance governor on Relforge, Cloudelastic, and Elasticsearch hosts - https://phabricator.wikimedia.org/T386860 [18:56:34] lunch, back in ~40 [19:46:12] back [20:04:48] Trey314159: I have a cirrus-opensearch arm64 image at docker.io/gmodena/cirrussearch-opensearch-image:1.3.20. You can swap it for the default `image: ...` line in the ElasticSearch's config recipe. [20:05:08] at startup (docker compose up) it'll nag about: java.security.AccessControlException: access denied ("java.io.FilePermission" "/usr/share/opensearch/.aws/config" "read") [20:05:27] but it's safe to ignore. [20:06:06] FWIW I've been using it to work on cirrus in past last day and seems to be stable enough [20:06:24] gmodena: Sweet! However, I got a 404 at the link above. Any idea what's up? [20:08:07] ah! docker (the command) will understand the URI, but your browser won't resolve it. [20:08:17] You can browse the repo/image at https://hub.docker.com/r/gmodena/cirrussearch-opensearch-image [20:09:22] and execute it locally with `docker run docker.io/gmodena/cirrussearch-opensearch-image:1.3.20`. It will fail at boot though, because of missing deps. Works fine if you start it from docker compose [20:09:52] FYi. The digest of this image should be sha256:24931ccf5f124dfecb302140faf5b58d3123da583a88e01ef9b9374ae0529b1c [20:20:40] gmodena: Cool! Thanks! I will try to test it this week.. though I just broken my current set up when I rebased earlier today. Ugh. [20:21:03] Trey314159 oh no :( [20:21:30] Eh, par for the course. [20:36:04] inflatador: ah sigh, somehow I did not spot there was a typo in the plugin name https://gitlab.wikimedia.org/repos/search-platform/opensearch-analysis-stconvert/-/blob/1.3.x/build.gradle?ref_type=heads#L37 :( [20:36:39] now that you mention it I remember Erik mentionning this typo... I can build another one if you prefer [20:47:19] dcausse no rush on that one [20:49:17] I think we're going to have to rebuild the plugins package pretty soon anyway based on https://etherpad.wikimedia.org/p/relforge-os-migration#L40 , but I could be reading that wrong [20:49:20] inflatador: no worries, the fix is easy the burden is mainly on your side to re-package a new deb file [20:50:31] inflatador: regarding that list I think it's managed by the puppet file doing the mapping, did you spot anything else being wrong other than this typo? [20:51:47] dcausse not yet, that was just a guess on my part that we were missing some of the plugins in the package. If it's just incorrect puppet code, that's good news ;) [20:52:41] the list is long enough that a typo is easy to make... [20:54:18] inflatador: actually, looking at /etc/opensearch/relforge-eqiad/opensearch.yml this mapping did not work at all [20:54:23] on relforge1004 [20:54:32] ??? [20:54:39] it has plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stconvert,analysis-stempel,analysis-ukrainian,cirrus-highlighter,extra,extra-analysis-esperanto,extra-analysis-serbian,extra-analysis-slovak,ltr [20:57:23] will release a new version of stconvert tomorrow, if that's ok with you pushing yet another deb file might be better to get rid of this typo sooner than later [20:57:53] sure, hit me up whenever. I think I found the problem w/the keystore https://gerrit.wikimedia.org/r/c/operations/puppet/+/1121087 [20:58:59] inflatador: thanks! not sure I understand all this yet but I'll take a closer look tomorrow [21:00:14] inflatador: mainly what I worry about is that we might be missing somethign with what Erik has done in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1091325 [21:01:05] going afk [21:02:58] ACK, see ya ytomorrow [21:09:29] oops, that's the wrong file anyway...good catch [21:16:01] I think Erik added that keystore module to clean things up a bit, but it doesn't seem to work with secret data like the old way does. Maybe something to do with how secrets are merged in? I'm gonna try to work around it for the moment [23:14:08] we fixed the keystore issue, it was a matter of changing the secrets repo path to match the role definition (using cirrus as opposed to opensearch). No changes needed on the Puppet side