[11:03:33] lunch [14:17:15] \o [14:37:52] o/ [15:10:01] eqiad being slow...codfw and cloudelastic have both moved on from commonswiki, but eqiad still reindexing [15:10:18] wonder if something is different there, cloudelastic is usually the slowest iirc [15:14:03] interesting [15:53:26] actually i guess it finished the reindex ~50 minutes ago, looks like it's syncing out replicas of the new index, but same diff [16:06:35] looks like we're getting a `CirrusSearchBackendMemoryIssue` for CODFW, checking now [16:06:44] :S [16:12:16] hmm, not actually seeing anything at https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1&from=2026-11-10T04:11:55.063Z&to=2026-11-10T16:11:55.063Z&timezone=utc&var-cluster=elasticsearch&var-exported_cluster=production-search&viewPanel=panel-9 [16:13:21] nm, it's under dnsdisc.memory_issue [16:20:04] looking at the graph, yea it's just a quick blip to a low value. maybe alert is too sensitive [16:20:35] curiously we are still rejecting ~10 req/s from search pool, even though we are in multiple datacenters now [16:21:10] i suppose reindexing might add some additional load, but probably not that much [16:24:15] looks like cirrussearch2108 tripped the "parent" breaker on the main cluster, checking secondary clusters [16:26:06] cirrussearch2094 tripped a bunch of times on psi as well, I guess I'll look at these hosts to start with [16:32:39] not seeing anything jump out in terms of pressure stalls or JVM memory usage [17:03:41] yea i'm not really sure. memory usage should mean jvm heap pressure, it would probably show up in the GC graphs [17:06:41] The alert fired and cleared again, will take another look when I get back (~45m or so). Maybe we have a node with some hot shards [19:07:26] dinner [19:31:19] hmm, tracked down yet-another intermittent cindy failure. Problem is backlink cache in core isn't reset between jobs, so later jobs in the same process can get a stale cache. Doesn't happen in prod, that runs 1 job per process, but CI runs many [19:40:09] looking into a couple options...not sure best way [19:59:57] Claude is teaching me some OpenSearch troubleshooting tricks, but this is still mostly looking like hot shards in CODFW [21:22:36] Claude is leaning hard into changing some GC settings, specifically `XX:G1ReservePercent=25` and `-XX:InitiatingHeapOccupancyPercent=30` . I wish I could share the chat but apparently we can only share stuff at a project level [21:23:27] hmm, my interpretation is it wants to preserve more memory for the young heap? I dunno, maybe not the worst thing to try but not sure what it will do [21:23:34] s/young heap/young generation/ [21:30:22] Yeah, it's saying that we wait too long before doing combined GC, although that doesn't really explain why CODFW is being bitten and EQIAD isn't [21:32:54] I started T444363 to kick it around. The hosts in that ticket all have big shards, but not sure how much more compared to other hosts [21:32:55] T444363: Cirrussearch: Investigate CirrusSearchBackendMemoryIssue alerts in CODFW - https://phabricator.wikimedia.org/T444363 [22:15:45] hmm, seems like we might have 2 reindexes of enwiki happening at the same time, per `curl -s "$H/_cat/indices/enwiki_general*?v&h=index,health,pri,rep,docs.count,store.size,creation.date.string`? [22:42:43] ah, the other reindex of enwiki failed, I'll delete that one [23:13:27] OK, updated T444363 with a few suggestions, feel free to take a look if/when y'all have time [23:13:27] T444363: Cirrussearch: Investigate CirrusSearchBackendMemoryIssue alerts in CODFW - https://phabricator.wikimedia.org/T444363