[01:15:08] ori: around? [01:26:03] jackmcbarn: hey [01:26:13] about to run off, but here for another minute or two [01:27:18] ori: does https://gerrit.wikimedia.org/r/#/c/227591/ make memcached only get hit if the local cache isnt there? [01:30:03] jackmcbarn: no; local means the server's cluster's memcache [01:31:40] :( ok. i have more questions but i'll save them for another day [01:32:12] jackmcbarn: AaronSchulz might be around. Or, if he isn't, just ask and I'll try to answer when I'm back on later. [18:40:03] Anyone know where class 'OOUI\apexTheme' comes from? It's not in vendor's master... [18:43:44] Ah, apparently monobook is broken with mw+vendor masters. [18:46:55] csteipp: update monobook to master [18:48:21] we made a change in the case of the class names [19:42:57] anomie: still there? [19:43:05] ori: Yes [19:43:58] anomie: (sec, gathering my thoughts :)) [19:45:51] anomie: re: https://phabricator.wikimedia.org/T102199 , it looks like the session loss errors are 'session_fail_preview', most likely from if ( $this->mTriedSave && !$this->mTokenOk ) { ... } in EditPage::getPreviewText() [19:46:07] they are getting logged from the non-api app servers [19:46:25] is it a consequence of the shortened expiry of tokens, and the lack of automatic retry logic for non-ajax requests? [19:48:57] Krinkle, https://en.wikipedia.org/wiki/MediaWiki:Gadget-HotCat.js :P [19:49:01] Thanks [19:50:14] ori: Are token expiries shortened? [19:51:00] anomie: that's me paraphrasing something timo just mentioned and that i may have misheard.. is that not the case? [19:51:16] And it'd be hard to automatically retry non-ajax requests without defeating the purpose of CRSF tokens, I think. [19:51:32] right, I'm not suggesting that that is a solution [19:51:40] just trying to isolate the problem [19:51:43] ori: I don't know anything about tokens being shortened, but that doesn't mean they haven't been. [19:54:20] anomie: hmm, ok! thanks [19:54:21] ori: Another possibility is if sessions aren't being refreshed as often to reduce cross-DC traffic, that could make sessions (and therefore tokens) expire more frequently. [19:56:52] anomie: hmmm -- we appear to be hitting this code path in User::matchEditToken() quite often: [19:56:54] 4176 if ( $val != $sessionToken ) { [19:56:54] 4177 wfDebug( "User::matchEditToken: broken session data\n" ); [19:56:54] 4178 } [19:59:14] ori: That happens whenever the session expired, User::getEditTokenAtTimestamp() generates a fresh one that obviously won't match the old one. [19:59:44] ok, so that is happening *a lot*. fuck. [20:00:25] anomie: where does the code that refreshes sessions live? [20:00:40] (thanks for answering my noob questions and for taking time away from whatever you were doing before i interrupted) [20:02:24] csteipp, ready when you are [20:03:08] ori: ObjectCacheSessionHandler [20:03:21] ori: I'm not sure about that one, possibly ObjectCacheSessionHandler (::write() and ::handleShutdown()) for us. In which case it may be that I4afaecd caused it. [20:05:37] I4afaecd sure looks suspect [20:05:48] What's the default expiry? [20:05:58] (maxage) [20:06:24] DefaultSettings.php:2282:$wgObjectCacheSessionExpiry = 3600; [23:02:27] anomie: looks like you were right on the nose: http://graphite.wikimedia.org/render/?width=688&height=385&_salt=1438383596.749&from=-6hours&target=divideSeries(MediaWiki.edit.failures.session_loss.count%2C%20MediaWiki.site.edits.count)&drawNullAsZero=true (I synced a revert of that change on 20:14 UTC) [23:04:21] hmmm... there should be a reasonable way to reduce the spurious session touches without causing that much dammage [23:04:32] sounds like a fun problem for AaronSchulz [23:36:03] does /w/extract2.php ring a bell for any of you? [23:36:09] it [23:36:22] it's used to pull a portal page from meta [23:36:29] but I can't figure out where it lives [23:36:43] https://en.wikipedia.org/w/extract2.php?template=Www.wikinews.org_template [23:36:47] gwicke: mediawiki-config [23:37:03] legoktm: ah, thx! [23:39:45] legoktm, could you protect https://meta.wikimedia.org/wiki/API_listing_template for me? [23:40:42] I'm not a meta sysop. [23:40:42] oh, okay [23:40:42] Katie is [23:40:42] thx again! [23:40:46] or find one in #wikimedia-stewards [23:40:47] np [23:43:12] MaxSem: mw.config.get( 'wgEventLoggingBaseUri' ); [23:44:03] ori, "//en.wikipedia.org/beacon/event" :O [23:44:22] possibly a race condition [23:44:43] the code is: [23:44:45] sendBeacon: !baseUrl [23:44:45] ? $.noop [23:44:45] : navigator.sendBeacon [23:44:47] ? function ( url ) { try { navigator.sendBeacon( url ); } catch ( e ) {} } [23:44:49] : function ( url ) { document.createElement( 'img' ).src = url; }, [23:58:10] bd808: well the handleShutdown() should already do that, unless something is preventing it from triggering [23:58:47] AaronSchulz: didn't we have to use afteR_post_send or something? [23:58:52] register_postsend_function or w/e [23:58:59] there was some issue with register_shutdown_function in HHVM [23:59:42] I wasn't aware of any special hhvm issue, but I can try testing around with it