[01:45:42] Tim-away: You've answered this question before, but I didn't write it down anywhere, so I have to ask you again: what is it about APC in HHVM that is substantially different and better than APC in PHP5? [01:46:20] HHVM is designed from the start to include a bytecode cache [01:47:26] whereas PHP5 does not really have bytecode, and storing its nearest equivalent of bytecode requires parsing a whole lot of structures and swizzling pointers [01:48:18] APC in PHP5 also does terrible things like takes a snapshot of the class list before including a file, then taking one after including, then examining the difference betwen the two arrays in order to work out which classes were added by the file [01:48:54] so it's basically nasty and inelegant, and for a long time it was crashy as a result [01:49:27] since HHVM was designed from the start to cache bytecode, it doesn't have all this nasty inelegant code [01:50:11] in PHP 5, APC does two things: bytecode caching and user data storage in shared memory [01:50:41] in HHVM, the core does the bytecode caching, and so the extension called "apc" is just the user data storage API [03:05:40] TimStarling: Yeah, that makes perfect sense. [03:11:30] TimStarling: I wrote a simple script to aggregate the Xenon logs by the topmost frame, to see which functions are on-CPU, regardless of call-path. And the results are unsurprising: at the top are DatabaseMysqli::doQuery, Memcached::getByKey, AutoLoader::autoload, Hooks::run, DatabaseMysqli::mysqlConnect, ReplacementArray::replace, CurlHttpRequest::execute, Elastica\Transport\Http::exec. [03:11:49] so the app servers are basically spending most of their time blocked on IO [03:12:21] most as in more than 50%? [03:12:28] I aggregated the second-topmost frame and found that Gadgets was using memcached very heavily [03:12:52] so I switched gadgets to use APC and the results have been nice: http://ganglia.wikimedia.org/latest/graph.php?c=Memcached%20eqiad&m=cpu_report&r=4hr&s=by%20name&hc=4&mc=2&st=1431486355&g=network_report&z=medium [03:13:02] re: 50% -- dunno, let's see [03:22:41] TimStarling: https://dpaste.de/O39J/raw [03:23:34] so less than 50% [03:25:13] I'm not surprised that mysql is at the top of this list, there's a lot of diversity in CPU dominated functions [03:25:27] but 13.8% is a bit more than I would have expected [03:25:57] although I have to adjust my expectations for the fact that HHVM is faster than PHP PHP [03:26:30] since CPU-bound tasks are so much faster now, you would expect waiting on things to get more important [03:27:14] right. [03:27:45] why are /srv/mediawiki/wmf-config/CommonSettings.php and /srv/mediawiki/php-X.XXwmfXX/LocalSettings.php so prominent? [03:28:44] LocalSettings.php is especially bizarre since it's just include_once( "/srv/mediawiki/wmf-config/CommonSettings.php" ); [03:29:35] CommonSettings.php is large and monolithic [03:29:36] you wouldn't expect to see that as the topmost frame unless xenon happened to snapshot activity at the exact instant it was parsing that one line [03:29:56] yeah, maybe there is some first-call initialisation inside HHVM or something [03:30:10] oh, maybe [03:30:31] or maybe it is still "in" LocalSettings.php while it is parsing and loading CommonSettings.php [03:31:06] but then we wouldn't see CommonSettings.php [03:31:27] only once the VM starts executing it [03:31:52] ohhh, I see. Yeah, that would make sense [03:31:56] there will be an initial phase of the require call which is attributed to LocalSettings.php [15:14:43] legoktm: is there a way to specify multiple handlers for the same hook in extension.json? [16:04:02] tgr: Hooks: { "Event": ["Class:Func1", "Class:Func2"]} [16:05:50] legoktm: that's what I tried but it seems to pass func2 as a parameter to func1 [16:06:07] hmm [16:06:55] should I file a bug about that? [16:07:00] yes please [17:15:30] bd808: Hey, so how's it going with logger stuff a.t.m.? [17:15:42] pretty good actually [17:15:53] What can I expect in logstash from mediawiki log groups and is it time to (re)-enable the extra stuff [17:15:59] logstash is back up and running at full strength [17:16:49] which extra stuff do you want to turn on? [17:16:59] the stacktraces for warnings? [17:18:13] stacktraces for fatals :| [17:18:13] bd808: yes [17:18:17] yes [17:18:21] all of the above and more [17:18:23] all the things [17:18:25] everything :P [17:18:30] pump it until it spills over [17:23:11] heh [17:23:52] I have a couple of pending puppet patches that will cut out some stuff we don't need. I might be good to get them in before adding a bunch more. [17:24:21] Krinkle: I want to work on psr3 logging in lyon so that might be a good time to talk [17:24:26] bd808: what are they? [17:24:57] ori: https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/puppet+branch:production+topic:logstash,n,z [17:25:22] dropping duplicate records and making the syslog stuff prettier [17:26:04] there's a mw-core patch that was merged too that will cut the jobRunner log volume in half [17:52:50] bd808: Huh. My mediawiki-vagrant wasn't working right, complaining about "Archives directory /vagrant/cache/apt/partial is missing." when trying to update dkms until I did `vagrant ssh` and created the directory manually. Random guess is that mismatched "guest additions" version meant it didn't have the shared directory /vagrant, and was choking while trying to use that shared dir so it could update the "guest additions"... [17:53:18] hmmm [17:53:41] I suppose that could happen [17:53:55] the guest additions bootstrapping process is a bit wonky [19:05:10] is guest filesystem faster than sshfs? I suppose it probably is but maybe not... at deviantART we had our own vagrantish setup ( I built it before I ever heard about vagrant) and it used sshfs to mount the host filesystem [19:13:36] twentyafterfour: dunno. I did find a vagrant plugin for sshfs -- https://github.com/fabiokr/vagrant-sshfs [19:13:57] there are a bunch of different file sharing options and the all suck honestly [19:15:43] The vagrant-ish system I built at Kount put everything on the VMs local filesystem. If a dev wanted to have access to the files they actually exported from the VM to the host rather than the other way around [21:16:55] bd808: legoktm btw, I keep forgetting to ask you, what should we do about https://github.com/wikimedia/composer-merge-plugin/issues/18 ? [21:18:12] do it I guess? [21:20:00] legoktm: by whom? [21:20:17] dunno [21:21:01] :) [21:26:46] bah, sorry, didn't see ya'll in -office [21:37:51] greg-g: find a "volunteer" [22:29:08] AaronSchulz: https://gerrit.wikimedia.org/r/#/c/210639/ ?