[03:06:59] <halfak>	 puppy: http://imgur.com/i9bOpnD
[03:21:43] <harej>	 puppy????
[03:38:27] <Emufarmers>	 :>
[03:38:37] <Emufarmers>	 halfak: I just saw https://www.mediawiki.org/wiki/Lyon_Hackathon_2015/Buddies You were trying to absorb my non-stafferness, weren't you? :o
[03:41:12] <harej>	 I'd be very interested in working with halfak, but I don't really develop software. It's also vanishingly unlikely I will be in Lyon since I will be in Berlin the week before.
[03:41:49] <halfak>	 Emufarmers, I was. 
[03:41:59] <Emufarmers>	 harej: that ought to make it easy, logistically.
[03:42:08] <harej>	 It's the awkward gap week.
[03:42:16] <halfak>	 The awesome travel/work week?
[03:42:28] <Emufarmers>	 Surely there are Europeans you can stay with
[03:42:36] <harej>	 I mean, like, I have a job or something.
[03:42:49] <harej>	 I don't know if my overlords will approve me working remotely from Europe for that long.
[03:43:17] <harej>	 ...depending on if they actually get a contract to me. They said they wanted to bring me on by early March. :-(
[03:43:50] <halfak>	 Ahh.  Yeah.  This is one of the benefits of working remotely full time. 
[03:44:09] <halfak>	 If I can work out a place to work from every day, I could probably work from just about anywhere. 
[03:44:18] <harej>	 I will be a remote worker too, but I don't entirely understand the terms of my work yet, plus I'll be new.
[03:44:26] <halfak>	 Some people at the WMF travel a lot. 
[03:44:34] <harej>	 "Remote
[03:44:47] <halfak>	 harej, gotcha.  Makes sense. 
[03:44:49] <harej>	 Remote yet local at the same time.
[03:45:02] <harej>	 My core competency is living in DC, as opposed to the other half of my full time equivalency.
[03:45:48] <harej>	 This means I can show up in their office however much they want me to and train their staff (which will be a big part of my job!)
[03:46:22] <harej>	 It's part time, so I can bill all my hours one week and take the following week off, but I don't want to stretch that system too thin (I will already be using it for Berlin and Mexico City)
[03:48:15] <halfak>	 It's hard to work from a conference/hackathon too. 
[03:48:23] <harej>	 Well, the plan is to not work ;]
[03:48:28] <halfak>	 :) 
[04:58:27] <YuviPanda|zzz>	 halfak: heh, I’m back working from beaches again :)
[05:10:40] <halfak>	 Hey YuviPanda|zzz!  Have a good vacation?
[05:10:45] <YuviPanda|zzz>	 yes yes
[05:10:54] <YuviPanda|zzz>	 still at beach tho :D
[05:20:51] <Deskana>	 halfak: Honestly I wish I could travel more. :-(
[05:21:29] <Deskana>	 halfak: I used to travel a lot in my time in academia and in my 1.5 years at the WMF I've only travelled for work once and it was to my bloody home country.
[05:39:45] <YuviPanda|zzz>	 Deskana: I'm on a beacccchhhhh
[05:40:09] <Deskana>	 YuviPanda|zzz: Sad to be leaving that life behind? :-p
[05:41:00] <YuviPanda|zzz>	 Deskana: who says I am... ;)
[05:41:04] <Deskana>	 Hah
[05:41:10] <Deskana>	 There are beaches in SF, true.
[05:41:16] <Deskana>	 They'll probably be arctic by your standards though
[05:45:24] <YuviPanda|zzz>	 Deskana: I could still travel around inside the US
[05:45:27] <YuviPanda|zzz>	 Deskana: in some form or way
[05:45:29] <YuviPanda|zzz>	 unsure how yet
[05:45:34] <YuviPanda|zzz>	 Deskana: first priority is to fix my hand
[09:32:17] <_Fremen_>	 hello everyone
[09:32:18] <_Fremen_>	 I want to ask about colwiz and other research management software, is this relevant to here?
[12:35:39] <YuviPanda>	 halfak: you might like: https://phabricator.wikimedia.org/T90534
[19:57:43] <ottomata>	 Ironholds: yo!  
[19:58:00] <ottomata>	 did you know...that page_id is now in x_analytics?
[19:58:20] <Ironholds>	 yep
[19:58:31] <Ironholds>	 I saw; it's a good change :). Hope y'all have fun with it.
[20:02:08] <ottomata>	 haha
[20:02:13] <ottomata>	 don't you CARE?!
[20:02:23] <ottomata>	 does it make pageview def easier?
[20:06:14] <Ironholds>	 ottomata, yes, but I'm not working on that in the long-term, sooo... ;p
[20:06:23] <Ironholds>	 like, it will become your problem or ellery's problem or aaron's problem or whoevers.
[20:06:30] <ottomata>	 dawwwww
[20:06:33] <ottomata>	 :)
[21:09:35] <Ironholds>	 halfak, leila, if you get a chance today could you do a 30s check on the dataset I emailed out about?
[21:09:53] <Ironholds>	 I'd like to get that out sooner rather than later (and then spend the weekend building a dynamic visualisation system around it. Booya.)
[21:13:25] <Ironholds>	 wb, DarTar 
[21:13:37] <Ironholds>	 your connection died; the only other thing I had was, if you have 30s can you check out the dataset I emailed about?
[21:13:50] <leila>	 Ironholds: I'm looking at the dataset
[21:14:07] <leila>	 I'm not sure if it's fine to share rows with very few pageviews
[21:14:17] <DarTar>	 Ironholds: sure
[21:14:20] <Ironholds>	 leila, example?
[21:14:29] <leila>	 first line of the table
[21:15:07] <leila>	 I understand that this table is based on sampled logs, so that may help you with entries with only few pageviews.
[21:15:26] <Ironholds>	 oh! it's a percentage!
[21:15:47] <Ironholds>	 1 == 1%
[21:15:55] <Ironholds>	 sorry, I should've made that clear in the email/table format
[21:16:14] <leila>	 I see. so it says: 1% of the pageviews from country x are to url/project y?
[21:17:01] <Ironholds>	 yup
[21:17:07] <Ironholds>	 or, sorry;
[21:17:14] <Ironholds>	 1% of pageviews to project y are from country x
[21:18:24] <leila>	 I see.
[21:18:28] <leila>	 I'll respond to the thread.
[21:18:43] <DarTar>	 what’s the subject of that thread?
[21:20:42] <halfak>	 Nettrom, did you email me the location of that wp1.0 labeled dataset?
[21:20:47] <Ironholds>	 thamls!
[21:20:51] <halfak>	 I can't seem to find the message.
[21:20:52] <Ironholds>	 ....thanks, even!
[21:21:12] <halfak>	 DarTar, "Research-Internal] Country/language pageviews informatio"
[21:21:14] <Ironholds>	 DarTar, "Country/language pageviews information"
[21:21:16] <Ironholds>	 snap
[21:21:17] <halfak>	 I win
[21:21:19] <halfak>	 :) 
[21:21:20] <Ironholds>	 nope
[21:21:24] <Ironholds>	 yours missed the trailing character
[21:21:32] <halfak>	 boo
[21:21:32] <DarTar>	 ok thanks
[21:21:41] <Ironholds>	 FAILS THE CHECKSUM. REJECTED PACKET. RESEND AT A PSEUDORANDOM INTERVAL IN LINE WITH THE VAN JACOBSON PROTOCOL
[21:21:48] <Ironholds>	 says my internal TCP/IP nerd.
[21:21:54] <halfak>	 Nettrom, nevermind.  I found it in my other inbox. 
[21:21:58] <Ironholds>	 (yes, I have an internal TCP/IP nerd)
[21:23:59] <leila>	 Ironholds: I checked out the table data briefly. I'm not sure if it's safe to release it or not given that I don't know of all the other data sets we're releasing.
[21:24:10] <Ironholds>	 leila, fair; thanks!
[21:25:22] <leila>	 my concern is for projects that are small, and then the percentage of pageviews can be large per country for those projects but the actual numbers can be small
[21:26:44] <DarTar>	 Ironholds: what’s the time period during which the data was aggregated?
[21:26:55] <Ironholds>	 DarTar, all of 2014
[21:30:48] <DarTar>	 Ironholds, leila: I really can’t think of any privacy concern if this is data aggregated over a year and given that these are percentages
[21:30:57] <halfak>	 And sampled!
[21:31:04] <DarTar>	 totally
[21:31:09] <Ironholds>	 and <1% == "aggregated as 'Other'"
[21:31:15] <halfak>	 :) 
[21:31:29] <Ironholds>	 I might suck at science but I'm a paranoid bastard around data
[21:31:32] <DarTar>	 the only edge case is the one in which this would disclose the country of an editor
[21:31:54] <Ironholds>	 I mean, for that to happen you'd need...
[21:31:55] <Ironholds>	 *thinks*
[21:32:05] <DarTar>	 I really can’t see how you would do that even combining all public data we release as of today
[21:32:26] <DarTar>	 brb
[21:32:40] <Ironholds>	 if you had a project with only 10 pageviews and a single editor from a specific country which has no readers, and you rolled a 1,000 sided die and landed on 0, that could happen.
[21:33:24] <halfak>	 Wouldn't we need to know the true pageview count too?
[21:34:24] <halfak>	 Did you guys see the public job application posted to analytics-l?
[21:35:40] <DarTar>	 halfak: no?
[21:35:46] * DarTar checking
[21:36:24] <DarTar>	 halfak: oh god
[21:36:37] <DarTar>	 I wish I could undo that :( 
[21:36:57] <halfak>	 It looks like he isn't looking to get paid. 
[21:37:19] <halfak>	 This looks like a "hey, where do I go to volunteer" email. 
[21:37:38] <DarTar>	 regardless, I think there’s a mistake about using a public mailing list for that
[21:37:43] <halfak>	 +1
[21:37:48] <halfak>	 Totally
[21:37:59] <Ironholds>	 halfak, yep! re true pageview count
[21:38:06] <halfak>	 I'm considering responding to the email and welcoming him to some of our public-facing projects. 
[21:38:09] <DarTar>	 so, I suggest we reach out to him and at least I remove the entry from the archives
[21:38:11] <Ironholds>	 DarTar, awesome! Could you note as such in email? That way I can get on to doing what I do best; geo data.
[21:38:19] <DarTar>	 halfak: that would work
[21:38:25] <Ironholds>	 I'm impressed he got mailman to accept attachments
[21:38:27] <DarTar>	 Ironholds: sure
[21:38:28] <halfak>	 DarTar, do you think I should reach out to him publicly or privately?
[21:38:30] <Ironholds>	 ta!
[21:38:35] <DarTar>	 halfak: privately please
[21:38:38] <halfak>	 Sure. 
[21:38:42] <DarTar>	 thanks
[21:40:34] <leila>	 DarTar: why should we remove the entry from archive?
[21:40:52] <DarTar>	 leila: because there is personal information incldued in that email
[21:41:01] <leila>	 I interpreted it as what halfak says, offering to volunteer.
[21:41:09] <DarTar>	 including an endorsement letter 
[21:41:23] <leila>	 ow! I just saw the resume, DarTar. 
[21:41:24] <DarTar>	 I think the intention was to reach out to us, not to the planet
[21:41:25] <leila>	 makes sense
[21:41:37] <halfak>	 leila, should I CC you on my response?
[21:41:38] <leila>	 haha! fair.
[21:41:53] <leila>	 nono, halfak. go ahead. I just wanted to learn why we're removing it from the archive
[21:41:57] <halfak>	 He might be more interested in your work than mine after all.
[21:42:00] <halfak>	 Oh sure :) 
[21:45:45] <Nettrom>	 halfak: if you want the dataset split into training/test-sets, I've got that for you too
[21:45:50] <Nettrom>	 just let me know
[21:46:09] <halfak>	 No worries.  My training scripts do all that for me.  I'm gathering features now :)  
[21:46:23] <halfak>	 It's really fast to gather features when you only need the text from the current revision. 
[21:46:40] <Nettrom>	 cool! I'm retraining my model, forgot to set the RF parameters correctly
[21:46:49] <halfak>	 When I'm scoring for vandalism, I need diffs, site information, the first revision to the page and the most recent revision from the user!!
[21:47:08] <halfak>	 What are the "correct" params?
[21:47:47] <halfak>	 Man... at this rate, we're going to have a working wikiclass model in revscores by the end of the day. :) 
[21:48:23] <halfak>	 Just so long as I can get my pull requests merged.  
[21:48:38] <halfak>	 I suppose I can always just create a branch, merge all the pulls to that and install it. 
[21:48:48] <Nettrom>	 params (besides criterion='entropy', which Wiki-Class already sets): n_estimators=501, min_samples_leaf=8, random_state=42
[21:49:16] <halfak>	 Anything meaningful to that random_state=42?
[21:49:18] <Nettrom>	 (although you can of course argue I shouldn't set the random seed)
[21:49:23] <halfak>	 :P 
[21:49:38] <halfak>	 Only if you're planning on cross-folding
[21:49:40] <Nettrom>	 helps to get repeatable results when you test, though
[21:49:44] <halfak>	 +1
[21:50:09] <DarTar>	 halfak: I don’t even think I can moderate mailman archives :-/
[21:50:17] <DarTar>	 post hoc, that is
[21:50:36] <halfak>	 Bah! 
[21:50:42] <halfak>	 There's got to be a way. 
[21:51:17] <DarTar>	 bwa-ha-ha: http://wiki.list.org/DOC/How%20can%20I%20remove%20a%20post%20from%20the%20list%20archive%20or%20remove%20an%20entire%20archive%3F
[21:51:30] <DarTar>	 “You need to edit the raw archive "mbox" directly,”
[21:51:32] <DarTar>	 right
[21:51:55] <halfak>	 yeah.. probably not. 
[21:52:10] <halfak>	 There should be someone at the WMF who has dealt with this before. 
[21:54:02] <DarTar>	 I’ll file an RT ticket if ops considers this worth the effort
[22:03:07] <halfak>	 For the sake of the curious, we asked around and it turns out that it is near impossible to remove a message from a list.  However we might have luck with removing attachements, so we'll reach out to ops to try that. 
[22:04:07] <ashwinpp>	 hi halfak, thanks for the mediawiki-utils library
[22:04:20] <halfak>	 :)  I'm glad it is helpful.
[22:05:13] <halfak>	 I'd like to know any suggestions or pain points you have as you work with it. 
[22:05:22] <ashwinpp>	 I started streaming over the dump, but it looks like it'll take much longer than 24 hours on 16 cores. I think I'm doing something wrong. Just wanted to check in about it.
[22:05:37] <halfak>	 Is it running on one of the stat machines?
[22:05:42] <ashwinpp>	 Yes 1002
[22:06:12] <halfak>	 Can you point me to your script?
[22:06:59] <ashwinpp>	 sure, give me a minute.
[22:07:44] <DarTar>	 hey ashwinpp, welcome to da channel
[22:07:57] <ashwinpp>	 thanks :)
[22:08:19] <leila>	 ashwinpp: I guess I owe an email to introduce you. DarTar is our research lead.
[22:08:28] <leila>	 halfak you know by now. ;-)
[22:08:40] <halfak>	 o/
[22:08:44] <halfak>	 :D
[22:09:07] <ashwinpp>	 Yes 
[22:10:10] <halfak>	 Code looks good. 
[22:10:37] <halfak>	 I'm looking at /home/ashwinpp/code/wikimedia/src/main/python/extract_link_diffs.py
[22:12:12] <halfak>	 ashwinpp, I'd be happy to take a pass on the code and run a couple of tests. 
[22:12:22] <halfak>	 ahh!  
[22:12:39] <ashwinpp>	 Oh, I thought I would have to push it on git :P
[22:13:01] <halfak>	 That would be great!  I'd just fork it and submit a pull request :D
[22:13:19] <halfak>	 So, I see one potential problem.  
[22:13:30] <halfak>	 I think that you are processing all of the bz2 and 7z files. 
[22:13:42] <halfak>	 This line: files = [x for x in os.listdir(head_path) if 'enwiki-'+date+'-pages-meta-history' in x]
[22:13:55] <halfak>	 Doesn't seem to specify an extension. 
[22:13:59] <ashwinpp>	 Oh okay. That was stupid
[22:14:12] <halfak>	 Na.  I do that all the time.  That's why I thought to look for it. 
[22:14:15] <halfak>	 :S
[22:14:39] <ashwinpp>	 I'll correct that.
[22:15:08] <ashwinpp>	 you can also check log.txt in the same folder
[22:15:51] <ashwinpp>	 if you grep for "finished" files, its at 46 right now after 12 hours
[22:16:27] <halfak>	 Yeah....  We should definitely be running much faster than that -- especially since you are skipping everything before last month!
[22:16:49] <ashwinpp>	 There are a total of 176 files I have to stream (I had mistakenly taken it to be twice the number)
[22:17:04] <halfak>	 +1
[22:17:05] <ashwinpp>	 So it looks like it should take ~50-55 hours
[22:17:21] <halfak>	 Some files are much larger than others too. 
[22:17:30] <ashwinpp>	 yes, that is also true
[22:17:39] <halfak>	 Those files contain pages like WP:Sandbox and WP:AIV
[22:19:12] <leila>	 ashwinpp: the bot has received some comments/questions. did you get a chance to go over them? 
[22:19:18] <ashwinpp>	 Oh, I see
[22:20:08] <ashwinpp>	 leila, yes I went over them and have the answers more-or-less ready. Bob will put up the comments after we discuss them today
[22:20:49] <ashwinpp>	 There are some suggestions which we should incorporate, like filtering out countries etc. so that we don't over-link.
[22:22:29] <ashwinpp>	 And there are some heuristics which we are using to find the correct anchor text for creating a link, but they can be tweaked and are up for discussion. However complicated heuristics are hard to justify.  
[22:23:53] <ashwinpp>	 halfak, the CPU usage also doesn't seem to be full utilized. Could disk I/O be the bottleneck?
[22:24:25] <halfak>	 It could.  CPU could also be a bottleneck on the decompressors. 
[22:24:56] <halfak>	 You end up using twice as many cores as you specify because it will load a decompressor separate from the multiprocessing.Process. 
[22:25:09] <leila>	 ashwinpp: makes sense. the comments are very thorough, which is nice, since they can help you improve it.
[22:25:56] <halfak>	 I'm not sure where the bottleneck would be here though.  It looks like the decompressors are not toping out either. 
[22:26:10] <halfak>	 It could be the log writes. 
[22:26:22] <halfak>	 That seems unlikely, but that's the best hypothesis that I have now. 
[22:27:15] <ashwinpp>	 leila: yes. I was pleasantly surprised to see such effort and detailed analysis. It actually set me thinking in different directions :)
[22:27:40] <leila>	 :-)
[22:28:45] <ashwinpp>	 log writes are 1 line per page at the maximum. The link diff writes are almost always more than that. 
[22:36:59] <halfak>	 ashwinpp, yeah, I'm wondering if you are getting poor performance from multiprocessing.Queue.  I have had that trouble before. 
[22:37:19] <halfak>	 It's that datastructure that collects all the results from each processor. 
[22:39:13] <ashwinpp>	 I see, but there is not a lot being handed over to it by yield
[22:39:55] <ashwinpp>	 I think one improvement, which I should do is to not store the last diff before the last month and work around it. This way I will reduce the overhead of reading the text before last month. 
[22:40:43] <ashwinpp>	 Sorry, by last diff, I meant the last revision text
[22:49:24] <ashwinpp>	 halfak, so should I run it with 8 threads?
[22:49:33] <ashwinpp>	 cpu_count()/2?
[22:52:21] <ashwinpp>	 Actually, I might be wrong about the log writes. I'll remove the unnecessary lines. 
[22:57:09] <halfak>	 ashwinpp, I'd run it with the full cpu_count.  Just make sure to NICE it. 
[22:58:28] <ashwinpp>	 sure, 20
[23:43:10] <guillom>	 Ironholds: Got a moment?
[23:43:25] <guillom>	 (not urgent)
[23:43:39] <Ironholds>	 guillom, totally!
[23:43:46] <guillom>	 cool
[23:56:42] <DarTar>	 hey halfak, I got a ping from the folks behind m:Research:Impact of Wikipedia on Academic Science, did we decide who would reach out to them in a volunteer capacity? I haven’t and I forgot if I was supposed to
[23:57:08] <halfak>	 Ack!  That was my bad.  I had planned to reach out to them. 
[23:57:12] <halfak>	 I'll get on that right away. 
[23:57:46] <DarTar>	 thx dude
[23:58:20] <DarTar>	 halfak: make sure you reply to both, I got the ping from Shane
[23:59:06] * Ironholds grumblegrumbles
[23:59:10] <Ironholds>	 I hate writing dynamic web apps