[10:47:11] lunch [12:29:43] \o [12:34:45] o/ [12:57:27] o/ [13:11:24] o/ [13:15:22] \o [13:16:00] o/ [13:23:48] Hiya [13:23:56] it seems like we have search problems [13:23:57] https://logstash.wikimedia.org/app/dashboards#/view/mwdebug1002?_g=h@d66e526&_a=h@9037fff [13:24:08] hmm, ooking [13:24:21] I can repro on mwdebug with https://wikitech.wikimedia.org/w/index.php?title=Special%3ASearch&search=GitLab%2FGitlab+Runner%2FTrusted+Runners&wprov=acrw1_0 [13:24:31] and also on prod (but not all the time) [13:25:19] atsukoito: loading the logstash link complains about error restoring state, wants to use the share link instad [13:25:54] it is on https://logstash.wikimedia.org/app/dashboards#/view/mwdebug1002 main [13:26:27] https://logstash.wikimedia.org/goto/fece8768bde3968ef8697e2465735313 [13:27:52] hmm, according to the code thats from some code in private settings, looks like it happens to be invoked by search. hmm [13:28:25] it actually can be not a search but a whole site, because I get it on main [13:28:30] that particular bit of code looks to be from stran on thurs, the 25th [13:28:54] yeah, it is on main, too.. so probably not search [13:29:01] not sure what i should say in public about the security tickets, but i would suspect the 22e6501 commit in private settings [13:29:20] but mostly only because it added the code [13:42:45] huh, haven't noticed this before: CirrusSearchTitleSuggestIndexTooOld [13:43:29] it triggered over the week-end because apiportalwiki was removed, I dropped related indices and it should recover [13:43:37] ahh ok, thanks! [13:46:36] it's actually quite rare that we remove wikis completely, usually we just "close" leaving read access open [13:54:47] ebernhardson: sorry for a jumpscare, i was thinking that it is something related to search since i probably hit the cache on the other pages, but it was not [13:55:39] atsukoito: no worries! it's always nice to check [14:58:29] Sorry for the short notice, but I have to deal with some family obligation and can’t make it to the triage on time. Might be able to join 30’ in. [15:00:33] kk [16:18:11] We have our first OpenSearch 2 host in CODFW prod [16:18:53] nice! [16:20:16] Note that I had to remove the local logstash due to JVM issues. I'm not sure how important that is, if y'all use those a lot let me know. I was going to suggest testing https://vector.dev/ as I believe Observability is looking at it as a possible replacement for rsyslog [16:20:59] inflatador: the local logstash was for log routing, it's how the logs got into kafka. I suppose check if the new server is still shipping logs [16:24:40] ebernhardson I'm guessing the OpenSearch logs are no longer pushed to Kafka with this change, or they're in a sub-optimal format. I'll check, but FYI some workarounds were discussed in https://phabricator.wikimedia.org/T324335#11809615 [16:35:50] Nope, things definitely seem broken based on the logstash Discovery interface. And that's true for the OpenSearch 1.x hosts as well. So it sounds like we do need to fix things [16:40:37] i suppose that suggests we rarely use those logs if they've been broken some time [16:42:59] Yeah, I think I uncovered this at the last migration and promptly forgot about it ;( . I'll see if there's a ticket and create one if not [16:56:11] OK, created T430567 [16:56:11] T430567: Fix logging pipeline for OpenSearch 2.x/Trixie hosts - https://phabricator.wikimedia.org/T430567 [16:58:33] I also see `MjolnirUpdateFailureRateExceedesThreshold: Data shipping to CirrusSearch in codfw` alerts are going off [16:59:12] oh mjolnir might still be sending the "_type" field? [17:01:19] Yeah, could. be. The alert just cleared FWiW [17:02:17] hmm, yea that seems plausible. checking [17:02:23] mjolnir uses an old elasticsearch client, that must be it? [17:04:27] yes it has elasticsearch=7.10.1 apparently, perhaps we could pause the search daemon in codfw while it gets fixed? [17:04:52] yea that seems reasonable. Not 100% sure how to turn it off, pause puppet and stop systemd? [17:04:58] interesting, I thought 7.x would be new enough not to send `_type` [17:05:39] But yeah, I can create a ticket for mjolnir as well and shut it down if y'all need me too [17:06:46] the es7 java client failed with opensearch2, I suppose the python client could be less strict, but to be fair it could well be transient failures, haven't checked the actual error [17:07:01] hard to say, reading the parallel_bulk code it seems like it's not really doing anything with the lines we pass it, should be pass-thru [17:07:38] there's a comment saying that the index and doc type should not be passed so hopefully it's not in the body [17:08:14] the es7 java client failure was in the client itself, an NPE trying to access "_type" in the response [17:08:37] it's been so long...trying to remember where to find mjolnir logs in logstash [17:08:43] i suppose it's good it's worked well for that long :) [17:10:05] hmm, so the bulk updater is pushing plenty of things now, there are only a few failures. So that implies it's not the client always adding them [17:10:44] but there are 4 batches of 10k failures, so it's semi-consistent [17:12:43] we hardcode _type: _doc in convert_to_esbulk [17:16:14] :( [17:16:39] so the data is already kind of borked :/ [17:17:13] yea, probably need to re-run the weekly [17:18:38] sure [17:19:28] have https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/80 but waiting on CI [17:19:53] dinner [19:56:24] ebernhardson looks like the Mjolnir alerts are still flapping, is that expected? Just wondering if we should suppress alerts [20:42:54] inflatador: yes just suppress them. The problem is the files containing the bad data are already generated and waiting to be processed, but the fix means new files will be generated correctly [20:43:22] probably mute for 2 days? [20:43:56] i think i can...poking around in alerts.wikimedia.org [20:47:05] ebernhardson no worries, I suppressed for the next 7 days [20:47:13] thanks!