[00:22:36] <wikibugs>	 3operations, Incident-20150205-SiteOutage, MediaWiki-Core-Team, Wikimedia-Logstash: Prototype Monolog and rsyslog configuration to ship log events from MediaWiki to Logstash - https://phabricator.wikimedia.org/T88870#1022054 (10bd808) 3NEW a:3bd808
[00:35:15] <wikibugs>	 3operations, Incident-20150205-SiteOutage, MediaWiki-Core-Team, Wikimedia-Logstash: Prototype Monolog and rsyslog configuration to ship log events from MediaWiki to Logstash - https://phabricator.wikimedia.org/T88870#1022085 (10chasemp) Sounds good!
[00:51:38] <wikibugs>	 3MediaWiki-Core-Team: Multi-DC documentation RFC - https://phabricator.wikimedia.org/T88666#1022132 (10aaron) https://www.mediawiki.org/wiki/Requests_for_comment/Master/slave_datacenter_strategy_for_MediaWiki
[01:10:21] <wikibugs>	 3MediaWiki-Core-Team, wikidata-query-service: Figure out if Neo4j is a possible alternative to Titan - https://phabricator.wikimedia.org/T88571#1022195 (10Smalyshev) Main problem with Neo4j so far is indexing - edge indexes are either manual addition (http://neo4j.com/docs/2.2.0-M03/indexing-add.html) or auto-in...
[01:28:15] <wikibugs>	 3MediaWiki-extensions-SecurePoll, MediaWiki-Core-Team: Set up mini wikifarm in Labs which has SecurePoll on it - https://phabricator.wikimedia.org/T88725#1022247 (10bd808) @anomie Does your existing securepoll testing setup in labs cover this need? I started to setup a new system using labs-vagrant but then noti...
[06:01:00] <wikibugs>	 3MediaWiki-Core-Team, wikidata-query-service: Figure out if Neo4j is a possible alternative to Titan - https://phabricator.wikimedia.org/T88571#1022449 (10Smalyshev) Moving to nodes didn't work either - when defining indexes for every property, it creates over 20K lucene files and tries to open all of them and g...
[13:52:16] <wikibugs>	 3MediaWiki-Core-Team, wikidata-query-service: Investigate ArangoDB for Wikidata Query - https://phabricator.wikimedia.org/T88549#1022562 (10Fceller) Maybe we can take a step back and ignore the ArangoDB specifics for the moment. I'm also organising NoSQL conferences and consulting NoSQL in general.  Still, I mus...
[14:25:23] <Nikerabbit>	 _joe_: http://stackoverflow.com/questions/18354845/how-to-debug-an-hhvm-server looks like someone else might be having the same issue
[14:28:46] <wikibugs>	 3MediaWiki-Core-Team, wikidata-query-service: Investigate ArangoDB for Wikidata Query - https://phabricator.wikimedia.org/T88549#1022585 (10Manybubbles) >>! In T88549#1022562, @Fceller wrote: > Maybe we can take a step back and ignore the ArangoDB specifics for the moment. I'm also organising NoSQL conferences a...
[14:38:44] <Nemo_bis>	 The answer there doesn't look very useful
[15:26:34] <Nikerabbit>	 _joe_: now hhvm stuck with admin server running... though I have no idea what to look in there. everything in vm-tcspace is under 20%, check-health says load 8, queued 254, check-ev says 524
[15:27:35] <_joe_>	 load 8?
[15:27:37] <_joe_>	 mmmh
[15:27:47] <_joe_>	 that's pretty low
[15:27:53] <_joe_>	 how many cores do you have?
[15:28:16] <Nikerabbit>	 _joe_: 4
[15:29:27] <_joe_>	 mmh ok setting hhvm.server.thread_count to something sane like 32 may help
[15:29:44] <_joe_>	 but I'm not sure at all
[15:30:04] <_joe_>	 we'd get more threads to debug and we'll see a patern at least
[15:30:22] <_joe_>	 Nikerabbit: I have to upload my slides on debugging hhvm to commons
[15:30:30] <Nikerabbit>	 _joe_: I saw them
[15:30:40] <_joe_>	 oh ok right, wikimedia :)
[15:31:06] <_joe_>	 sorry, back to rugby :)
[15:32:06] <Nikerabbit>	 np
[15:53:48] <Nikerabbit>	 now stuck with 34
[16:01:00] <_joe_>	 Nikerabbit: take a stack trace
[16:01:07] <_joe_>	 and paste it somewhere
[16:01:25] <_joe_>	 maybe we can try to nail this down finally
[16:04:47] <Nikerabbit>	 yeah just a moment
[17:28:48] <Nikerabbit>	 _joe_: https://translatewiki.net/static/temp/hhvm-stack.txt
[17:56:17] <_joe_>	 ok, classic hhvm deadlock
[17:56:31] <_joe_>	 but, it's on we-don't know what
[17:56:52] <_joe_>	 maybe something like some destructors fuckup
[17:57:41] <_joe_>	 to be clear, this happens on our cluster as well, but like once/day over all the 250 servers
[17:57:46] <_joe_>	 or less
[18:21:18] <Nikerabbit>	 so we are back to square zero, perhaps with the difference that we now know what we can reproduce in matter of minutes?
[18:34:01] <_joe_>	 well, I'll try to reach our fb friends 
[18:34:16] <_joe_>	 did you by chance try with their latest package?
[18:35:00] <Nikerabbit>	 _joe_: I didn't know I was expected to
[18:35:11] <_joe_>	 oh, not really
[18:35:22] <Nikerabbit>	 I've been using WMF packages for long time
[18:35:25] <_joe_>	 hold on anyways
[18:35:51] <_joe_>	 ok I think it's better; after all, we test them on mediawiki quite well :)
[18:36:48] <Nikerabbit>	 yes
[18:36:53] <Nikerabbit>	 _joe_: thanks for the help again
[19:33:53] <wikibugs>	 3MediaWiki-Core-Team, wikidata-query-service: Investigate ArangoDB for Wikidata Query - https://phabricator.wikimedia.org/T88549#1022926 (10Smalyshev) >>! In T88549#1022562, @Fceller wrote: > Still, I must admit that I'm not familiar with the internal data model of Wikipedia. I've checked with George Washington...