[07:53:06] volans|off: I went ahead and depooled cirrussearch2113 and downtimed it for 12 hours if you'd like to proceed with the firmware stuff [07:53:32] cluster's happy and in green state so you can muck with it without issue [08:06:39] ryankemper: hey, thanks a lot, doing it right away so we get this out of the way :) [08:24:39] ryankemper: all done, thanks a lot, the host is all yours [08:28:34] awesome! [13:14:52] \o [14:44:36] random awkwardness: enwiki search for `america` matches `United States` via the `AmericA` redirect when `America` would be slightly nicer. [14:45:13] probably not worth thinking much about :P But hadn't thought before about how there are even stylistic choices when choosing the redirect to display [15:03:19] meh...of course. My test query `am;erica` was only being fixed by m1 (Query similarity). Until i used it...then m0run (session similarity) saw the clickthrough and is now making the suggestion [15:38:36] I'm out for the weekend! Reminder: Monday is a public holiday in Switzerland. [16:51:41] ryankemper heads-up that I'm going to roll-restart EQIAD, this'll be the first time we've used the cookbook with 100% openstack hosts [17:03:32] err...OpenSearch hosts that is [17:09:52] ack! [17:11:18] The main cluster is unhappy, due to master change. Checking on it now [17:17:20] OK, we're back [17:17:47] just needed to restart opensearch on a couple of the new masters, I thought I did that already but I guess not ;( [17:20:40] starting the rolling operation again [18:53:12] rolling operation is working well so far [19:08:47] spoke too soon! Another quorum issue [19:09:52] all good, this one fixed itself [19:12:17] OK, the rolling operation completed successfully! [19:20:26] great! and glad the restart caught that not every new master had been restarted [20:13:59] yeah, that's a good feeling for sure. We still need to roll back the cookbook reimage stuff (ref T391350), but things are moving in the right direction [20:13:59] T391350: Review/rollback all temporary changes from the OpenSearch migration - https://phabricator.wikimedia.org/T391350