[07:39:01] investigating "Found unexpected field names page_type,redirect_target for page 1735377 in arzwiki" [07:39:22] the producer appears to still be sending schema version 1.1.0 [07:40:24] might need a restart I guess? [07:46:35] restarted staging and it now produces 1.2.0, will do eqiad & codfw [07:58:11] ok restarted the rest and seems to have fixed the logspam, not sure how much time we lost with this for populating the new redirect fields :/ [08:00:19] we get few update failres because of the circuit breaker since opensearch 2, filed T431784 [08:00:19] T431784: Cirrus update: Unknown error 429 circuit_breaking_exception - https://phabricator.wikimedia.org/T431784 [08:19:49] sigh... the schema upgrade did not go well, unsure why... [08:22:09] restarted consumer-cloudelastic from a fresh state [08:24:32] consumer-search in codfw is affected but not in eqiad... [08:37:58] not the first time we bump the schema to add a new field... something related with the state or something new with flink2... [10:22:53] lunch [13:06:35] we got a question in #wikimedia "how is wikipedia running OpenSearch 2.19.5 when CirrusSearch only supports version 1.3.x?" ... sounds like we might need to update Cirrus docs? [13:06:43] o/ [13:07:43] the master branch supports 2.19 indeed but I don't think there's an official release supporting it yet, looking [13:15:10] Re: Checking for active reindexes, it is just a `curl _cat/tasks" | grep -i index`? [13:18:19] dcausse: I'll proceed with reviving the circuitbreaker alert T431827, seems like we had it in the past, and it is the second time it would have indicate that something is going on in last 2 weeks [13:18:24] T431827: cirrussearch: alert for tripping circuitbreakers - https://phabricator.wikimedia.org/T431827 [13:29:19] \o [13:30:49] .o/ [13:32:52] o/ [13:33:48] inflatador: thanks [13:34:24] on our side I think we need to handle those gracefully but yes if there's a particular why they appear now it'd be great to know why [13:34:39] s/particular/particular reason/ [13:58:48] i feel like this used to be faster...the commonswiki reindex on cludelastic says running for 17h, ETA 2d 22h [13:59:19] eqiad has 19h running and 10h remaining, codfw 21 running and 8h remaining [13:59:27] maybe cloudelastic is just that slow [13:59:30] ebernhardson we lost about 4 hrs when I rebooted a server yesterday ;( [13:59:56] inflatador: sure, but big clusters says 8-10h remaining, cloudelastic says ~70h :) [14:00:25] makes cloudelastic ~3x slower than the big clusters [14:00:56] inflatador: also it's not a big deal to reboot, in theory the orchestration sees the fail, cirrussearch cleans up the index (most of the time), and then it retires. Loses a little time, but not a big deal [14:00:57] touché. Is that a new thing or has it always been this slow? [14:01:14] i'm not sure, i feel like it used to be closer to the same but we don't have concrete metrics anywhere [14:02:08] inflatador: oh and you had asked about how to check status, the (very rough) doc is the repo readme: https://gitlab.wikimedia.org/repos/search-platform/cirrus-reindex-orchestrator [14:02:53] basically whoever runs it should have a directory in their home dir named cirrus-reindex.$DATE. From there `venv/bin/python3 -m cirrus_reindexer.report eqiad/state.json` gives a status update [14:02:56] --verbose prints a little more [14:03:08] on the deployment host [14:04:52] ACK, will add to our docs [15:07:50] dcausse, inflatador: here is a alert for circuitbreakers, https://gerrit.wikimedia.org/r/c/operations/alerts/+/1309685 [15:08:29] looking [15:33:55] thanks! [16:08:04] atsukoito dcausse looks like the new alert is firing, is there a runbook for this? [16:08:30] no :) [16:09:01] LOL, well ya gotta start somewhere ;) . I imagine there are many things that could trigger this? [16:09:59] yes, almost all endpoints might be hitting this I guess, we'll have to go down the rabbit-hole of understanding what circuit breakers are actually triggered [16:10:28] indexing is one of them, but search is likely to hit this too [16:12:13] could it be reindexing? [16:14:05] ah perhaps? [16:16:16] not clear if they're visible to us when using the re-index module [16:16:20] * inflatador should probably look at fixing our logstash pipeline sooner rather than later [16:16:31] yea i'm not sure either [16:17:13] yes having opensearch logs would be helpful [16:20:21] I was hoping to use vector.dev but otto-mata has a suggestion in T324335 that might be worth a look [16:20:22] T324335: Remove logstash from the CirrusSearch servers - https://phabricator.wikimedia.org/T324335 [16:22:29] yes... opensearch must be able to ship ecs logs [16:22:39] heading out, have a nice week-end [16:22:45] i haven't looked cloesly but it must be able to log to syslog and get forwarded like everything else in our infra. The logstash bit was, IIRC, just some bandaid we applied around the age of elastic 1.x [16:25:47] https://opensearch.slack.com/archives/C051JEH8MNU/p1769058171657979 [16:25:54] that was the last I looked at it [16:26:03] inflatador: i'm looking on how to work with this alert now :) [16:26:15] I'm sure there's a way to do it the ECS thing, just need to spend a little time [16:31:19] if it's just a log4j formatter, i'm sure we could write up something in a day or two and stick it in search-extra [16:31:34] since it wouldn't have to meet any random persons needs (like an upstream patch) it can probably be trivial [16:32:24] or we could even just apply the existing patch from the ticket and fork the formatter into search-extra [16:42:37] ebernhardson that could be viable if you think it wouldn't take too much time. I've been playing around with vector.dev in my homelab and I did get it to produce ECS logs, but I'm not sure what or who needs to approve mirroring the vector repo and all that stuff [16:59:41] inflatador: i'll take a look at it, seems reasonably easy [17:04:52] ACK, thanks! [17:10:22] i suppose this does add some potential for extra cpu costs...it looks like (from the abandonded PR) the problem with upstream is that they, effectively, printf and pretend that counts as json formatting. The PR uses a real json formatter [17:10:47] maybe there is an intermediate "escape this json string" option [18:31:45] i have a plausible log formatter based on the abandoned PR, using jackson streaming. It's basically a fresh implementation of the interface, but pretty simple [18:38:08] i should have realized before..wikidata is convenient when checking a version migration since it puts every possible language into one index. At least in theory it would find errors initializing various langs [18:38:20] * ebernhardson noticed because wikidata was failing :P [18:38:35] localy [18:56:32] nice! [18:56:56] I suppressed the circuit breaker alerts until Tuesday [19:00:29] Also, it looks like we got a `consumer-search@eqiad is failing write requests because of unknown errors` alert that flapped recently, see #operations. Guessing it is the same thing d-causse is already investigating so I suppressed alerts 'till Monday [19:02:53] not too sure what those are, the writes should just get auto-retried so not a big deal at these rates i think [19:47:46] far too sus...locally have cirrus integration suite passing with all plugins against opensearch 3.7.0 [19:52:45] cirrus needed almost no changes. changed the version constraint an nGram -> ngram