[15:12:48] _808db: about? [16:03:24] Reedy: he's on vacation [16:03:39] He's breaking the rules! [16:04:50] legoktm: ping [16:04:59] hoo: hi [16:05:22] legoktm: Did you ever test the centralauth part of mw.ForeignApi in production? [16:05:54] no, just on my test wiki [16:05:55] I see it requesting a CentralAuth token... and using it [16:06:02] yes... [16:06:06] but wikidata's API says that the token is invalid [16:06:14] errr [16:06:25] example request you're making? [16:08:18] mh... now it works, but it also no longer includes the CA tokens [16:09:44] I'm getting the invalid token error... [16:09:57] if you're logged in on wikidata it won't use the token [16:10:01] ah ok, so it checks whether one is logged in [16:10:04] I see, yes [16:10:07] that's smart [16:10:22] MatmaRex: ^^ [16:10:50] hm [16:11:04] we also recently changed the backend store for the tokens [16:11:16] the client seems to get this all right... I guess somethings not quite right on the backend [16:11:21] yeah [16:11:27] I deleted my wikidata.org cookies, and running (new mw.ForeignApi('https://www.wikidata.org/w/api.php')).get({action: 'query', meta:'userinfo'}).done(function(data){console.log(data)}); fails from en.wp [16:11:31] is it maybe being partitioned? [16:11:48] So that WD can't see the token from enwiki? [16:12:31] huh [16:13:01] hoo: legoktm: i can reproduce. the same thing works with commons.wikimedia.org though [16:13:27] looks backendy to me. and i have no idea how the backend for this works [16:14:00] I presume that there are different redis shards involved, thus the tokens aren't really shared cluster wide anymore [16:14:03] if I delete my commons cookies, it fails there too [16:14:16] hoo: but we don't fragment the cache by wiki... [16:14:22] AaronSchulz: ^ [16:14:32] But maybe by some kind of user dbname combination? [16:14:34] Or some such [16:17:32] $key = CentralAuthUser::memcKey( 'api-token', $loginToken ); [16:17:40] which just prefixes with $wgCentralAuthDatabase [16:19:42] mh [16:22:06] https://twitter.com/librariesio [16:22:15] lol, that makes Paladox redundant [16:23:16] Reedy: not until it can upload patches to upgrade the libraries and auto-rebase them! [16:25:12] legoktm: CentralAuthUser::getSessionCache [16:26:59] So it uses nutcracker... and then I have no idea [16:27:23] I did some eval.php tests and its broken in the backend [16:28:17] mediawiki_session_redis_servers in hiera defines the servers to use [16:28:35] is redis's BagOStuff::add() behavior different? [16:28:55] CentralAuthUser::getSessionCache()->add( $key, $data, 60 ); [16:28:59] why do we use add there instead of set? [16:29:55] anomie: ^ you wrote this? [16:37:47] legoktm: I don't remember why it might use add instead of set. [16:38:35] my eval.php testing shows that add() appears to work fine [16:38:57] legoktm: Ah. The data stored shouldn't change, so there's not much point in overwriting it if it's already set. [16:39:25] ok [16:43:57] mh... the more I look at it, the less ideas I have [16:49:34] legoktm: I have an idea [16:49:35] did it just break today? [16:49:38] oh [16:49:45] Look at CentralAuthHooks::onApiCheckCanExecute [16:49:57] $centralUser = CentralAuthUser::getInstance( $user ); [16:50:01] And then it checks the idea [16:50:03] * id [16:50:17] How should that work? [16:50:54] it checks the ids of the central users... [16:51:13] yes, but how should the $centralUser be populated [16:51:16] how did that ever work [16:52:18] oh hrm [16:52:29] well....that code hasn't changed since ever [16:54:31] legoktm: Hmm. If I hit https://en.wikipedia.org/w/api.php?action=centralauthtoken&format=json&servedby=, then ssh into the server that actually served the request, eval.php it, and var_dump( CentralAuthUser::memcKey( 'api-token', $token ) );, it still doesn't work. [16:56:40] o.O [16:57:00] so redis backend is broken? [16:57:52] Well, eval.php tells me that $wgMemc is an instance of MemcachedPeclBagOStuff, not RedisBagOStuff. [16:58:20] anomie: Use CentralAuthUser::getSessionCache(); [16:58:28] That should be a different object [16:58:54] hoo: That's not what's being used for the API centralauthtoken feature. [16:59:02] er, we no longer use $wgMemc [16:59:12] anomie: git pull? ;) [16:59:37] anomie: https://gerrit.wikimedia.org/r/#/c/234839/ [17:00:42] Ah. Hmm. [17:04:03] also, we did the switchover to redis last week [17:04:09] so why did it start failing today? [17:06:49] logins broken? [17:07:45] csteipp: I hope not... but the centralauthtokens for cw api use are [17:12:28] legoktm: Ok, I see that ->add() is failing on RedisBagOStuff for some reason. Now to figure out why. [17:39:10] MAybe AaronSchulz can help out here? [17:40:12] * AaronSchulz was reading backscroll [17:43:43] legoktm: I'm seeing "read error on connection" coming back from redis. [17:44:02] in response to the call to $conn->multi(). [17:44:05] nutcracker issue then? [17:44:35] is it a specific backend? [17:45:24] Some commands work fine, e.g. ping(). Others do this. Do we have some sort of command whitelist set up in our redis? [17:50:51] no whitelist afaik [17:54:03] ->ping() works, ->time() doesn't, ->info() doesn't, ->echo() doesn't. And, critically, ->multi() doesn't. [17:56:16] multi() may be a temproxy bug [17:56:46] for info() it at least makes sense that it doesn't work due to the proxying/hashing [17:57:19] * AaronSchulz noticed that before [17:57:58] https://github.com/twitter/twemproxy/blob/master/notes/redis.md [17:59:11] Almost sounds like we need an extra MW class a wrapper [17:59:54] multi() is not supported there, so that explains [18:00:03] CA can clearly just use set() for the tokens for now [18:00:35] I'll make a patch for that, maybe core, and poke at cas() later [18:01:04] Well, there we go then. RedisBagOStuff doesn't fully work with twemproxy because it doesn't support multi/exec, watch/unwatch, or info. [18:01:31] AaronSchulz: Thanks [22:13:00] tgr: I was thinking SpecialMWOAuth::handleAuthorizationForm would just throw an exception if $requestToken==''. Otherwise $oauthServer->getConsumerKey( $requestToken ) isn't really correct either. [22:38:48] AaronSchulz: did the api token stuff get fixed? [22:44:12] in master, I don't think hoo deployed it though [22:44:33] it's really our setup that is broken, but the CA change works around it [22:44:54] https://gerrit.wikimedia.org/r/#/c/238527/ is more complete, but blocked on zend stuff [23:13:17] AaronSchulz: ok, I'm backporting it