[07:24:05] errand [08:08:25] good morning! [08:35:15] hi! [09:11:31] we've got CirrusSearchNodeIndexingNotIncreasing on an upgraded OpenSearch 2.0/trixie node.. checking what's going on here before proceeding [09:16:08] and we generally got non-zero amount of new errors https://w.wiki/Rv2D [09:17:47] looking [09:18:17] CirrusSearchNodeIndexingNotIncreasing could be expected if the node does not have any shards yet [09:20:48] cirrussearch2087 seems dead so expected [09:21:31] from sal: log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cirrussearch2087.codfw.wmnet with OS trixie [09:29:27] yeah, seems like brian marked cirrussearch2087 as done, but it was one of the hosts we did in the beginning when the puppet was failing and we never fixed it, fixing it now [09:29:39] thanks! [13:00:27] \o [13:03:58] o/ [15:01:23] a full cindy run is about 15m (vs 25m before pre-loading the corpus), the cucumber suite passes in 2m40 * 2 os versin, 5m10, that means around 10m spent on various preparation steps [15:02:00] that is both awesome, and very sad. 2m40s is insanely good, 10m setup hurts :P [15:03:09] i've wondered how much of that setup could be parallelized...but for example when i tried to parallelize wiki creation it hit race conditions in mwcli (iirc it was the parts that rewrite ~/.config/mwcli/mwdd/default/ [15:03:10] yes... I'll check again what we could optimize there [15:31:23] something i've pondered but never worked out..it seems like one could initialize everything and then snapshot the running containers into a pre-configured setup. But then we still have to run mw database updates, and cirrus mapping updates, to catch any delta between the images and master. But that adds moving pieces and inconsistent starting points [15:31:51] sounds like a fun project though ;) [15:33:41] yes... my worry with snapshoting is that many things could change outside of our control... and I'm afraid that debugging inconsistencies will be painful [15:33:55] completely agree, the debugging situation just sounds too painful [15:34:20] will add some more info about the timings in the logs [15:35:36] but possibly some cleanups could be done async in the while wait loop, does not help when there are many patches to run but at least you could save the destroy phase [17:10:42] if only "mwcli docker mediawiki install" did not run install.php & cirrus main scripts we could possibly run those in parallel... [17:21:39] hmm, off the top of my idea no great ideas on how to work around it :S [17:30:14] yes... already dropped some unused containers (eventlogging/redis), I can at least use your trick to prepare the php image with ghostscript installed that'll save ~15s, other optimizations I could think of are possibly saving a couple ./mw docker env get/set but these takes ~1s so won't gain much... [18:53:32] going to guess CI is temporarily broken and try again in 30 min: PHP Fatal error: Cannot redeclare class Wikibase\Lexeme\Tests\MediaWiki\Scribunto\WikibaseLexemeLibraryTestCase [18:55:55] not clear where it would have come from, no recent changes to those repos [18:56:45] actually reading the log closer, it looks like autoloader tried to include the same file twice :S [19:37:54] we're getting a P95 alert for comp_suggest . Not sure what to make of that since the seemingly-depooled CODFW is affected https://grafana.wikimedia.org/goto/cfqq5hhqhdwqoc?orgId=default [19:42:01] hmm [19:43:01] it shows both datacneters climbing, thats quite high [19:43:52] based on DNS resolution and conftool, CODFW is depooled. But I don't have the tell on the percentiles dashboard anymore since we switched to DNS discovery. Or if we do, I'm not sure where to look. [19:45:44] i mean that graph shows mw@codfw->dnsdisc and mw@eqiad->dnsdisc both showing elevated, dnsdisc is eqiad since it's depooled [19:46:19] Looking at elasticsearch percentiles in grafana can also see it's complaining, first step i suppose restart 1086 since it's struggling, but not clearly the initial source [19:46:48] everything started climbing ~15 minute ago [19:46:57] ACK, on it [19:47:02] although curiously not really top-line qps. That looks stable [19:49:10] OK, 1086 is restarted [19:52:11] could plausibly be headed in the right direction, graphs are flatlining/turning, but probably too early to say [19:52:23] Yeah, that's what I'm seeing too. Hmm [19:56:11] yup, looks like alerts cleared [19:57:29] hmm, yea looks to be clearing up. I suppose my suspicion is since all traffic is at one cluster, and this is the peak time of day, but it's a hint of peak-load problems? Just once isn't enough to be worth doing anything about though [20:19:44] that seems plausible [20:29:53] oh my isn't this fun...so wikibaselexeme test suite has conditional class declaration. Essentially it has two different definitions of a class in the same file depending on if certain other extensions are also loaded [20:31:03] I don't fully understand whats going on...some optimization in the php engine i imagine, but after reproducing the CI failure locally can fix it by putting the second class definition in an else clause [20:31:40] as in, before it was `if (foo) { class blah { }; return} class blah { }`. and it hits duplicate class definition errors. but as `if (foo) { class blah { } } else { class blah { } }` it works [20:31:58] * ebernhardson thinks conditional class declaration might be partially insane either way :P [20:36:22] lol..."Of course, PHP picked the third way: “Why don’t we have both?” Now we have a complicated system that only works some of the time." - https://www.npopov.com/2021/10/20/Early-binding-in-PHP.html (npopov is a well known php core contributor) [21:45:12] ChatGPT gave a nice one-liner for getting migration process with the nodes API: `curl -s https://search.svc.codfw.wmnet:9243/_nodes?filter_path=nodes.*.name,nodes.*.attributes.hostname,nodes.*.os.pretty_name ` [21:45:16] ` [21:45:22] ` | jq -r '.nodes | to_entries | group_by(.value.os.pretty_name) | map({os: .[0].value.os.pretty_name, count: length}) | sort_by(.os) | .[] | "\(.count)\t\(.os)"' codfw-nodes.json` [21:45:50] oops, you can ignore the last `codfw-nodes.json`, that's from a static file ;P [21:46:11] `24 Debian GNU/Linux 11 (bullseye) [21:46:11] 28 Debian GNU/Linux 13 (trixie)` [23:25:09] While debugging `cluster.routing.allocation.exclude._name` not working (root cause ended up being us already having a transient entry for that, setting it to `""` rather than null), I found some other stale transient cluster settings left over on a few prod search clusters [23:25:39] I cleared the transient settings from `eqiad/codfw X chi/omega/psi`. Most were empty allocation excludes or values already covered by node config/defaults (`action.auto_create_index`, `cluster.routing.use_adaptive_replica_selection`). codfw chi also had transient logger overrides (`logger.io.netty`, `logger.org.elasticsearch.http`, `logger.org.elasticsearch.transport`, `logger.transport`, plus index-engine/allocation loggers) [23:25:58] I cleared those too since they looked like stale debugging state. Does anyone remember those logger overrides being intentional? If so, they should come back via persistent settings and/or Puppet config rather than transient, but I'm guessing those were probably added a couple years ago for some debug type effort and we just never circled back to clear those out