[01:13:01] <halfak>	 Hey folks!  We have a mailing list: https://lists.wikimedia.org/mailman/listinfo/ai
[01:13:03] <halfak>	 Sign up!@
[01:13:11] <halfak>	 lzia, ^ 
[01:13:27] <halfak>	 ToAruShiroiNeko, ^ 
[05:01:59] <halfak>	 Refactored features and datasources complete and tested. 
[05:03:40] <halfak>	 Now I just need to finish the languages refactor (mixins) and re-implement the basic datasources in APIExtractor.  
[05:03:46] <halfak>	 g'night!
[13:52:13] <Amir1>	 o/ halfak 
[13:52:15] <Amir1>	 http://pastebin.com/1GFAMSzk
[14:00:44] <Amir1>	 halfak: https://github.com/wiki-ai/editquality/pull/4
[14:00:46] <Amir1>	 Also this
[14:03:10] <Amir1>	 lzia: hey, if you're around. I built this based on your recommender system :) http://tools.wmflabs.org/dexbot/tools/recom.php?user=Ladsgroup
[14:22:40] <lzia>	 very nice, Amir! :-)
[14:22:49] <lzia>	 happy to see that it's already being used.
[14:23:55] <lzia>	 Amir1: are you pulling the randomly generated articles from en -> fa?
[14:24:41] <lzia>	 (brb, need to eat something and get ready to catch the train)
[14:25:47] <Amir1>	 no, I get last 20 articles of the user and recommend based on that 
[14:25:55] <Amir1>	 I need to go too
[14:25:57] <lzia>	 ah! got it.
[14:26:04] <Amir1>	 o/ 
[14:26:07] <lzia>	 we'll chat later, Amir. ciao ciao
[14:27:39] <Amir1>	 ciao :)
[15:56:51] <Amir1>	 o/ halfak 
[15:56:57] <halfak>	 Hey Amir1 
[15:57:05] <halfak>	 Saw your pull but haven't reviewed yet
[15:57:17] <Amir1>	 great
[15:57:19] <Amir1>	 no rush
[15:57:25] <Amir1>	 I tested some parts
[15:57:54] <halfak>	 Looks like you changed the call signature.  It turns out that mwxml can process many XML files in parallel with the map() function. 
[15:58:11] <halfak>	 So you can have <dump-files>... and then pass the list of paths to mwxml.map()
[16:00:52] <Amir1>	 hmm, let me take a look at it
[16:01:22] <halfak>	 Amir1, http://pythonhosted.org/mwxml/map.html
[16:01:54] <halfak>	 Note that you can pass paths to the mapper are compressed. 
[16:02:10] <halfak>	 The mapper will handle raw xml, gz, bz2 and 7z
[16:02:27] <halfak>	 It will call out to p7zip on the system for 7z though so that'll need to be installed. 
[16:03:02] <halfak>	 https://github.com/mediawiki-utilities/python-mwcli/blob/master/mwcli/files/functions.py
[16:03:57] <Amir1>	 okay :)
[16:04:03] <Amir1>	 I will make it work :)
[16:04:20] <Amir1>	 in several hours but right now I need some rest
[16:04:33] <Amir1>	 once it's merged I need to start writing the paper
[16:05:06] <Amir1>	 halfak: overall, do you like it?
[16:05:35] <halfak>	 Yes. I'm kind of thinking of just taking the next pass on it myself. 
[16:05:42] <halfak>	 Amir1, what do you think?
[16:06:27] <Amir1>	 good :)
[16:06:36] <Amir1>	 I like it
[16:06:58] <Amir1>	 I'll finish this 
[16:07:01] <Amir1>	 then you start
[16:09:07] <halfak>	 OK sounds good. 
[16:25:23] <Amir1>	 halfak: I'm trying to write that part but it seems I can't send arguments to process and in callback 
[16:25:25] <Amir1>	 https://github.com/mediawiki-utilities/python-mwxml/blob/master/mwxml/map/map.py
[16:25:40] <Amir1>	 should I change it works or change this file ^
[16:25:48] <Amir1>	 *the way
[16:25:55] <halfak>	 Amir1, what is the problem?  I'm confused. 
[16:25:59] <halfak>	 that function isn't a callback. 
[16:26:03] <halfak>	 It yields out values. 
[16:26:18] <Amir1>	 def page_info(dump, path):
[16:26:31] <Amir1>	 I want to send arguments to this function
[16:26:46] <halfak>	 Oh.  Just use a clojure. 
[16:26:54] <halfak>	 *closure
[16:27:18] <halfak>	 Variables that are in scope when you define the function continue to be available when it is called. 
[16:28:29] <halfak>	 https://en.wikipedia.org/wiki/Closure_(computer_programming)
[16:29:16] <Amir1>	 I used closures in javascript before
[16:29:20] <Amir1>	 but never in python
[16:29:26] <Amir1>	 I can understand how it works
[16:53:08] <Amir1>	 halfak: I'm done now :) check the new amend on my commit
[17:25:19] <halfak>	 Will do
[17:26:13] <halfak>	 Amir1, did you test?  It looks like that won't work. 
[17:28:21] <Amir1>	 Yeah, I tested it
[17:28:38] <Amir1>	 halfak: how did you test it?
[17:28:47] <Amir1>	 testing all aspects of the code is hard
[17:28:57] <halfak>	 Did you run it on multiple dump files?
[17:29:27] <halfak>	 So, the dump_processor function is what gets parallelized. 
[17:30:15] <halfak>	 See an example dump processor here: https://gist.github.com/halfak/c7a6bb267fcefb3aa14c
[17:30:30] <halfak>	 You want to yield a minimal amount of data from the processor function. 
[17:30:33] <halfak>	 brb
[17:52:34] <Amir1>	 hmm, when I give two dumps and it got crazy
[17:53:15] <Amir1>	 I know I should yield processed data but I couldn't make it happen
[17:53:24] <Amir1>	 can you give it a try halfak?
[17:53:38] <halfak>	 Will do :) 
[17:54:51] <Amir1>	 thanks
[18:42:12] <aetilley>	 SpaceX lands a rocket!
[18:42:14] <aetilley>	 http://www.nytimes.com/2015/12/22/science/spacex-rocket-landing.html?_r=0
[18:42:17] <aetilley>	 huzzah!
[18:48:01] <halfak>	 Pretty cool, I agree :) 
[18:51:14] * aetilley arrived in PA.  Should be on later.
[19:37:06] <ToAruShiroiNeko>	 halfak is this the future of commuting?
[19:38:00] <halfak>	 Hmm.  Probe-ably not. 
[19:46:54] <halfak>	 Also, I've learned that mixins are stupid and confusing. 
[19:46:57] <halfak>	 So none of that
[19:47:00] <halfak>	 More namespaces!
[19:52:33] <harej>	 Yes! More namespaces! Project! File! Module! Schema! Topic!
[19:53:57] <halfak>	 heh. 
[19:54:03] <halfak>	 So I'm making namespaces for classes of features. 
[19:54:25] <halfak>	 So we have bytes, temporal, wikibase, and wikitext
[19:54:43] <halfak>	 wikitext has some sub-namespaces: edit, parsed and tokenized
[19:55:08] <halfak>	 These are not wiki namespaces, mind you