[08:41:43] hiya! [09:08:20] hi! [09:15:40] i'm investigating unassigned shard check on omega @ eqiad, T431254, might ping you later [09:15:41] T431254: OpenSearch unassigned shard check on omega @ eqiad - https://phabricator.wikimedia.org/T431254 [10:12:14] lunch [10:58:16] lunch, too [12:51:44] \o [12:53:26] o/ [13:10:10] o/ [13:56:56] hiya!! [16:02:37] dinner [16:02:41] dcausse: ebernhardson: I forgot to ask: Is there anything we can do to enable multi-DC search infra for the second phase of semantic search? IIRC at least parts of the infra (inference?) where only single DC and cause a temporary outage during phase 1. [16:03:30] hmm, does dpe-sre have a k8s cluster in codfw? Does it have capacity? [16:03:53] otherwise, the answer is probably post-os 3.x upgrade when we figure out if we can unify vectors into the main cluster [16:04:10] (and tangentially, maybe we can push comp suggest out into a dpe-sre k8s cluster) [16:05:01] but only if there is a codfw cluster i suppose :P [16:13:27] ebernhardson good news, we do have dse-k8s-codfw and the current semantic search is deployed there now [16:18:14] in theory that should resolve concerns then, as long as mediawiki can query the dc-local instances [16:20:48] from a search perspective we should be OK going active/active, I think we were active/active for a short period but rapidly had to drop codfw because liftwing does not serve the embeddings in codfw [16:22:36] oh! i forgot about liftwing [16:23:17] but if we force the opensearch remote model to hit eqiad and not the discovery endpoint we can serve vectors from both eqiad & codfw [16:23:31] I might have done that already tho... checking [16:27:07] * ebernhardson sighs: UnexpectedValueException: $wgDBservers has dbname='cirrustestwiki' but $wgDBname='enwiki'. S [16:27:35] after updating mw and all my extensions to latest master [16:28:02] :/ [16:28:06] re: liftwing differences between CODFW and EQIAD, Dawid (ML SRE) is working on a solution for that. Not by adding embeddings but having some kind of service catalog to make it more obvious what's available from different instances. Will check for the ticket [16:33:03] https://phabricator.wikimedia.org/T422253 [16:39:11] thanks for the link I was searching for it [16:39:41] and I did not update the opensearch remote model to hit eqiad, they still point to inference.discovery.wmnet:30443 [17:03:28] and based on T422253 conclusion I think we should point to inference.eqiad.svc.wmnet, so no redundancy for the embedding service [18:56:13] hmm, mjolnir needs to update away from bullseye in CI. But newer debian doesn't have openjdk-11...probably ok but i wish we weren't drifying from prod [19:04:39] Interesting, since we have openjdk 8 on Bookworm [19:08:43] In Puppet world, it looks like we can get OpenJDK 11 on newer Debians, at least in theory https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/java.pp#19 [19:09:07] But I'm not sure what they translates to in Docker world, maybe we need a new image? Feel free to file a ticket if so [19:16:15] not sure yet..turns out mjolnir is going to be a pain to update :P The newer debian image also changes the python version, so it fails in multiple ways [19:18:18] updated jvm causes some error related to scala-maven-plugin...will be fun [19:28:36] * inflatador sighs [19:44:39] it might be as little as archiva going away and needing to switch it to maven central...but it's a very slow build so takes time to find out [19:53:09] sigh...we use spark 3.1.2 in prod, and that doesn't support java 17 :S maybe we use one of the off-brand jvm's for CI :) [19:53:28] (off-brand jvms are all just openjdk but somehow different, like the ones from amazon) [20:10:49] should stick to java8 no? not sure why this got bumped to java11 because we never ran java11 in the analytics cluster [20:12:22] it was already using java 11, i can check 8 as long as i'm swapping it [20:20:14] looks like we're red in eqiad chi, checking it now [20:20:46] :S [20:21:55] quite a few unassigned, the missing primary is commonswiki_file shard 9 [20:23:31] for commonswiki, there are a few options. Easiest is of course to re-add a node with the data. If not we have data loss and a few ways to handle [20:26:52] I'll check and see if I can bring back cirrussearch1111, if not I'd say we tell the cluster to accept old shards and requeue updates, does that sound reasonable? [20:27:09] can try that, although i suspect it doesn't have old shards [20:27:47] shards 11 and 16 of the same index are showing primary failed as well [20:29:27] yikes, you're right, I don't see any replicas or primary for 9 [20:29:35] i see 11 and 16 as having primarys according to apis, and i see recoveries runnning [20:30:55] but 9 has lost the primary. If we can't get the data back from disk i think we have to tell cluster re-route to accept an empty primary. Saneitizer will start fixing it, it's 1/30th of the data lost. We could potentially pipe the cirrus dumps for commonswiki into it and backfill commonswiki with streaming updater [20:31:14] assuming traffic is routed to codfw? [20:31:43] ebernhardson yeah, EQIAD is depooled ATM [20:32:44] i suppose piping the cirrusdump directly would be awkward, we would need to write a piece that rewrites it into super_detect_noop to avoid running revisions backwards [20:33:00] otherwise, there is also the snapshot api. Cloning codfw -> eqiad [20:33:19] snapshot sounds elegant, but i don't know about cross-dc or how long it takes [20:34:33] Looks like recovery from eqiad isn't gonna work [20:35:15] I'll start on the snapshot stuff, practicing against cloudelastic first. the last thing we wanna do is bork CODFW while trying to fix EQIAD [20:35:26] yup :) [20:37:40] wonder how much is still setup, /_snapshot responds with plausible info but i haven't used it in a long time [20:39:56] nope, fails verify. looks like it might nede some adjustments to know its not inside AWS [20:44:33] yeah, I don't think we've touched it since Elastic days ;(. I just created T431348 , will update when I have more progress [20:44:34] T431348: Restore lost index to EQIAD - https://phabricator.wikimedia.org/T431348 [20:47:28] +1 [21:03:43] OK, I found the value for `profile::opensearch::s3_password` , gonna try and follow https://docs.opensearch.org/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/#amazon-s3 [21:04:51] if S3 doesn't work we might try doing it as a shared filesystem and rsyncing it over [21:23:20] OK, I got a snapshot started on cloudelastic. per ChatGPT we needed to add a region to our snapshot repo definition first [21:50:56] https://docs.google.com/document/d/1TjKmUnOJT1tw_JVwNdyYT8IE0aLd5g4z9x9YSzyKRFA/edit?tab=t.0 working on restoration plan here [22:09:42] snapshot/restore successful [22:09:58] sorry, we restored a small index as a test [22:17:24] ebernhardson we're about to create the snapshot, looks like it's gonna be `commonswiki_file_1764596737` on CODFW. We're in https://meet.google.com/fde-tbpf-wqh? if you wanna join [22:23:40] OK, snapshot created [22:23:53] `curl -XGET "https://search.svc.codfw.wmnet:9243/_snapshot/elastic_snaps/snapshot_t431348/_status?pretty"` if you wanna check status [22:35:01] ebernhardson: we're realizing that snapshot restore onto a cluster that's on split opensearch versions might be a no-go (i'm not sure if it's prohibited but it def adds a lot of complexity). i'm thinking the allow empty primary shard + saneitizer approach might actually be saner (pun intended). thoughts? [22:45:12] OK, I told the cluster to accept an empty primary. We have data loss but should be able to move forward with the migration (albeit more carefully this time) [23:06:30] had to kill and restart the snapshot with lower throughput settings