[03:14:37] Krenair: "main stash" contract/intent per doc in ObjectCache is like a persistent and replicated version of Memcached [03:14:44] Definitely not like APC [03:15:20] Which means internally keys could be distributed, but either way should get the same result from all calling servers [03:15:53] And in practice afaik we don't even distribute stash/redis, only for memcached [03:16:05] And for memcached it's consistently hashed [03:16:33] If you get a different result from different app servers for keys that haven't change recently, that seems like a bug [03:25:42] Krinkle, okay [03:25:51] Krinkle, try logging into deployment-tin and deployment-mediawiki04 [03:25:55] and running =ObjectCache::getMainStashInstance()->get( "enwiki:captcha:894537469" ); [03:26:47] baring in mind that this key would have been created within the last few minutes [03:27:03] (though is still consistently appearing on one server but not the other) [03:27:13] Krenair: Im mobile at the moment [03:27:22] Ican look on Monday [03:27:27] ok [03:27:30] Or file a task If not already [03:28:22] It's possible nutcracker is miscondifured or somehow outdated on a sever [03:28:36] The reason I've dug into this is that sometimes, the form on beta enwiki's signup page has no captcha after reloading [03:28:58] it returns HTTP 400 ('Requested bogus captcha image' text) due to the mediawiki server not seeing that key [03:29:55] nutcracker.yml is identical between -tin and -mediawiki04 [03:33:44] created https://phabricator.wikimedia.org/T173646 [03:33:55] but 100% sure which are the correct projects yet, will need some investigation [03:34:20] s/but/not/ [04:06:43] Krinkle, given your explanation and the fact that 'keys *' on the two redis servers returns significantly different results, I'm thinking stuff is broken