[13:42:05] https://github.com/wikimedia/mediawiki/blame/master/RELEASE-NOTES-1.27#L9-L10 [13:42:10] As of 1.27, MediaWiki now requires PHP 5.5.9 or higher. This corresponds with [13:42:10] 10 HHVM 3.1. [13:42:15] Is that supposed to be 3.11? [13:42:58] 3.12 says 5.6.99-hhvm [13:43:12] Considering we only have 3.6.5 installed on our servers, probably not. [13:43:43] I don't presume 5.5.9 corresponds with 3.1 though [13:44:01] Further down it says... [13:44:02] MediaWiki 1.27 requires PHP 5.5.9 or later. There is experimental support for [13:44:06] HHVM 3.6.5 or later. [13:45:50] * anomie has no idea which versions of HHVM are supposed to be equivalent to PHP 5.5.9. [13:46:11] PHP_VERSION in 3.12 says 5.6.99 [13:46:16] So 3.10 or 3.11 might make sense [13:46:22] I'll file a bug, as either way, it's confusing [13:47:02] So does 3.6.5. [13:47:16] really? lmfao [14:44:17] Reedy: on the other hand MediaWiki spews fatal errors on diffs with 3.11 and 3.12 [14:44:25] lol [17:16:46] bd808, tgr, greg-g: I just submitted patches to fix https://phabricator.wikimedia.org/T125313. It'd be nice to get those backported to wmf.13 and wmf.14. [17:43:53] anomie, Reedy: The HHVM<->PHP correspondence is imperfect anyway. I found things that were introduced in PHP 5.4 but are missing from 5.6.99-hhvm [18:13:28] HHVM is missing things from 5.3 as well. It's a mixed bag especially for non-core language features (e.g. dba_* functions) [19:11:16] bd808: If https://gerrit.wikimedia.org/r/#/c/271028 looks right, I'll swat it today [19:36:26] csteipp: tgr reminded me that the authmanager channel is what makes the graphs at https://grafana.wikimedia.org/dashboard/db/authentication-metrics [19:36:41] So maybe we don't need them logged to disk as well? [19:37:24] See https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/logging.php#L145-L154 [19:37:46] bd808: Hmm... let me see if that has everything. How does that get into grafana, btw? [19:38:01] yeah, I overcomplicated that a bit [19:38:19] there is a custom logger for that channel in WiikimediaEvents [20:09:38] tgr, bd808, anomie: there's an increase in keys that corresponds to the regression [20:09:46] could you investigate? [20:09:48] * ori will be back in ~20 [20:11:33] ori: I don't think we've touched anything about how the User class cache works. [20:15:31] That key prefix corresponds to User::getCacheKey() [20:26:27] RoanKattouw: Yeah, so we shouldn't claim correspondence... Maybe say PHP X or HHVM Y etc [20:37:01] Sure. What makes that annoying is that HHVM's version numbers do suggest correspondence :? [20:37:03] * :/ [20:38:27] Yeah [20:39:05] bd808: is the logstash traffic spike the duplicate log messages? [20:39:35] Using .99 just feels like a hack [20:46:59] ori: bd808: speaking of dupes I have seen messages with tag: syslog, es,es # i.e. two es [20:47:04] and definitely duplicated message [20:47:06] (on prod) [20:57:19] ori: I can't tell by that time axis. Does it match Tue, Feb 9, 22:00? [21:00:03] bd808: yes [21:01:04] that change should have at most added 256 bytes to MW messages, but it would be easy to turn off and see if the graph dives down [21:01:22] assuming that the graph being up is a problem as well I suppose [21:28:54] is there a better way to deep clone stdClass objects in PHP than unserialize( serialize( $obj ) ) ? [21:49:10] legoktm: Write a function that recursively traverses the object and does clone() on every subobject? [21:51:13] https://github.com/myclabs/DeepCopy [21:51:59] bd808: I don't know that it's a problem; I'm down to looking at interesting shifts in traffic trends that coincide with the regression, which is admittedly a very unsophisticated way to debug. [22:12:04] ori: I looked at that, the code was overly complicated for what I need. I'm going to implement anomie's suggestion for now [22:12:11] nod [22:23:42] anomie: did the auth stuff touch centralauth:user memcached activity at all? because that spiked [22:28:48] ori: Only in that it creates CentralAuthUser objects much like the old code did, which will load them from memcache. [22:29:39] they're getting loaded about 10 times as often [22:35:20] users are loaded when the session is validated, to make sure that the token is still valid [22:36:06] how much sessions are used could have changed by sessionmanager, although I can't think of anything specific [23:15:00] ori: does static.php need sessions? [23:15:12] if it doesn't, disabling that is the first thing I would try [23:15:31] tgr: disabling that == disabling static.php [23:15:33] ? [23:15:42] no, session handling in static.php [23:16:05] but the regression started on the 9th, and static.php had not rolled out yet afaik [23:18:51] which regression? [23:19:13] for centralauth user lookups you only have the weekly comparison, right? [23:20:51] yes [23:24:08] ori, tgr: One thing that comes to mind is that pre-SessionManager, if nothing in the request unstubbed $wgUser then UserLoadFromSession would never be called which means CentralAuth probably wouldn't have reason to create a CentralAuthUser. index.php always unstubs, but I don't know how many load.php or api.php calls do. With SessionManager, CentralAuth will always be creating a CentralAuthUser during Setup.php for every request with valid CA [23:24:09] cookies, since it has to do that much to decide if the request has a CA session. [23:24:16] tgr: I don't see anything session-related in static.php [23:25:57] anomie: hmm -- how could we test this? [23:26:02] ori: which means it will initialize sessions [23:26:15] it calls setup.php and does nothing to explicitly disable them [23:26:30] ori: another credible candidate is http://tools.wmflabs.org/sal/production?p=0&q=wgAuthenticationTokenVersion&d= [23:26:59] if you look at the raw save timing graph, it's self-healing: [23:27:00] https://graphite.wikimedia.org/render/?width=925&height=556&from=00%3A00_20151205&until=23%3A59_20160215&target=MediaWiki.timing.editResponseTime.p75 [23:39:21] tgr: could you add this information to the task? [23:43:10] anomie: got time to look at the comment on https://gerrit.wikimedia.org/r/#/c/270999 ? [23:43:18] before the swat I mean [23:43:38] otherwise I can merge and backport as is, it's a nit [23:44:13] tgr: Yeah, you're probably right. I'd rather be explicit about setting it true though. [23:48:46] anomie: already 6 patches in today's swat [23:48:56] wanna pick one of your patches over the other? [23:49:18] or just backport them to .14? [23:49:40] tgr: If you have to pick, pick the CA one. [23:49:46] ack [23:53:02] anomie: are you around for swat? having to wait for 6 other patches means another hour without lunch :) [23:53:58] tgr: I'll probably be online. [23:54:20] cool, don't mind if I put your name down then? [23:56:05] Go ahead [23:56:25] thx