[06:02:30] hey folks, i'll be doing an es4 master switchover in about an hour T257847 [06:02:31] T257847: Switchover es4 master from es1020 to es1021 - https://phabricator.wikimedia.org/T257847 [13:05:33] cdanis: s10 (wikitech) can be set RO by dbctl like any other section? [13:06:27] oh god [13:06:52] maybe, I'll get back to you [13:15:54] cdanis: thanks :* [13:37:18] <_joe_> marostegui: I *think* it should. [13:40:48] _joe_: Yeah, I guess so too, as there is no way to do it on the current db-eqiad.php anymore [13:40:59] I will wait for Chris to confirm, there is no rush though [13:41:11] <_joe_> marostegui: try! [13:42:04] haha [13:43:09] yeah, that is right [13:43:31] <_joe_> https://www.youtube.com/watch?v=0VkrUG3OrPc [13:43:33] I got confused between labswiki and labstestwiki (s11) [14:06:15] elukey: do you know if anything like distributed healthchecking (with consensus) has ever been discussed for mcrouter? [14:09:20] cdanis: not that I am aware no.. do you mean consensus to take actions like failover to the gutter? [14:09:26] precisely :) [14:10:39] I didn't find anything in the issues tracker, didn't bother looking through the Facebook group where apparently development is discussed [14:11:05] it would be something of a design change; I think right now mcrouter only talks to memcacheds and each mcrouter is pretty independent [14:11:20] yep [14:12:05] but I think that the mcrouter upstream devs assume that it is fine to have this independence, given what should end up on memcached [14:12:13] now I know that in our case it is different :D [14:12:22] yeah I have the feeling our usage is somewhat atypical [14:18:10] in theory any state that we keep on memcached should be migrated to a more suitable tool, leaving only keys/values that can be recomputed without any impact [14:59:27] agree, that would make our lives a lot easier -- but I think it isn't going to happen any time soon [15:05:35] <_joe_> it is atypical in some ways [15:05:42] <_joe_> including the fact we have so many proxies [15:05:50] <_joe_> mcrouters, I mean [15:06:32] <_joe_> but still, of course, mcrouter doesn't assume global consistency in any form is a goal [15:06:53] <_joe_> btw, I am pretty sure our software works well without those expectations [15:07:17] <_joe_> proof is it worked for years with nutcracker, which is by design completely inconsistent [15:09:52] rzl: o/ - speaking of memcached, I am wondering if we should think about experimenting with NVMe before the eqiad server refresh (that should happen this fiscal IIUC) [15:10:37] nvme / on-disk cache is supported from 1.6.x onward, and we have 1.6.6 imported on buster-wikimedia (for IDP) [15:10:41] https://memcached.org/blog/nvm-caching/ [15:10:54] it may reduce a lot our footprint in term of shards etc..