[17:10:26] <halfak>	 hey Ironholds 
[17:10:47] <halfak>	 :)  How's the RfC stuff going?
[17:12:11] <Ironholds>	 pretty good! Just arguing with google
[17:12:15] <Ironholds>	 they don't like automated searching
[17:12:21] <Ironholds>	 which is, you know. Humorous.
[17:12:25] <Ironholds>	 how goes you?
[17:14:02] <Ironholds>	 halfak, ^
[17:14:31] <halfak>	 Not bad.  I'm still hanging out with family, so I might suddenly disappear.  
[17:14:51] <halfak>	 But I'm picking the revscoring project so that I can run some tests. 
[17:15:09] <Ironholds>	 cool!
[17:15:14] <Ironholds>	 I've been working on some UDFs
[17:15:32] <Ironholds>	 ran into a weird issue with getting maven to recognise JUnit, though, which has blocked me :(
[17:15:52] <halfak>	 Junit == unit testing framework
[17:17:21] <Ironholds>	 yup
[17:20:49] <halfak>	 Gotcha.  
[17:21:05] * halfak picks up ipython notebook for the 6th time. 
[17:21:14] <halfak>	 Let's see what's new. 
[17:21:23] <Ironholds>	 hehe
[17:33:07] <danilo_>	 hi halfak, I runed a test using your code http://pastebin.com/MmyKTAa5 , but it keep running like i was in a infinite loop and don't renponds (I don't know the word in English for that)
[17:33:37] <halfak>	 hey danilo_, not sure what's up.  I'll take a look in a few minutes. 
[17:33:58] <danilo_>	 ok
[17:41:10] <halfak>	 danilo_, I think I know what's up.  
[17:41:22] <danilo_>	 I have read a guide about support vector classification http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf and it says in item '2.2 Scaling': "We recommend linearly scaling each attribute to the range [-1, 1] or [0, 1]", and we have features like user_age_in_seconds with huge numbers, I think maybe that is the problem
[17:42:26] <halfak>	 So, I added probability=True to the constructor of SVC -- that allows us to ask for predict_proba()
[17:42:40] <halfak>	 But the docs say that the classifier will take substantially longer to train. 
[17:42:49] <halfak>	 Regarding the scaling, I think that you are right.  
[17:42:52] <halfak>	 We have two problems. 
[17:43:30] <danilo_>	 I tried without probability=True and I had the same problem
[17:43:41] <halfak>	 One is that a lot of these variables are long-tailed.  It would probably be better if we log-scaled them. 
[17:45:35] <halfak>	 danilo_, not sure then.  As you can tell, the examples work.  Could it be that you are training with substantially more observations than before?
[17:45:40] <halfak>	 How long did you let it run?
[17:46:21] <danilo_>	 2 hours
[17:48:11] <halfak>	 danilo_, I'm working on running my own tests.  I'll let you know if I end up having the same issue. 
[17:49:50] <danilo_>	 ok, I'll try to normalize all features to range [-1, 1] and test again
[17:57:45] <Ironholds>	 oh, halfak: about an hour after our discussion yesterday, my christmas presents from my secret santa arrived
[17:57:50] <Ironholds>	 they include a book on computing and automata ;p
[17:59:16] <halfak>	 Cool!  Which books?
[18:09:04] * Ironholds looks
[18:09:13] <Ironholds>	 Kozen's "automata and computability"
[18:15:37] <Helder>	 halfak: Hi!
[18:15:38] <Helder>	 halfak: Will it be possible to override the "hardcoded" params 'kernel="linear", probability=True' on
[18:15:38] <Helder>	 https://github.com/halfak/Revision-Scoring/pull/19/files#diff-f66d67fa60b7e7f300f42d45343c331fR16
[18:15:38] <Helder>	 ? I assume yes. In that case, will it be possible *because* of the kwargs which is *after* these parameters? (i.e., it will allow these parameters to be redefined because the last ones specified wins)
[18:16:04] <halfak>	 If you duplicate an arg, it will error out. 
[18:16:42] <halfak>	 But I am a fan of working with different kernels. 
[18:16:57] <halfak>	 It's just that this scorer is called "Linear" SVC, so that should probably be hardcoded. 
[18:17:10] <Helder>	 indeed :-)
[18:17:14] <halfak>	 We might want to make a simple "SVC" scorer that does not hard code such things. 
[18:17:52] <Helder>	 indeed² and the use it for the linear one?
[18:18:15] <Helder>	 (use it = subclass/instantiate) 
[18:20:06] <Helder>	 halfak: even if the "linear" should be hardcoded, what about the probability=True? Shouldn't we be allowed to toggle it?
[18:20:36] <halfak>	 Well, that's essential for returning probabilistic scores. 
[18:27:28] <halfak>	 Helder, I think it is certainly worth running some tests with different args, but we can do that by using the SVC model directly. 
[18:29:47] <Helder>	 makes sense
[18:30:48] <Helder>	 halfak: should I wait until this is fixed before merging? https://github.com/halfak/Revision-Scoring/pull/19/files#diff-478fa498e644949992e7ed83f5836dc0R37
[18:30:54] <Helder>	 "Doesn't work yet"
[18:32:10] <halfak>	 Nope.  it doesn't work yet
[18:32:16] <halfak>	 It will work in the other pull request. 
[18:32:25] <halfak>	 I split them up because the work was unrelated. 
[18:33:07] <Helder>	 ah, ok
[18:33:19] * Helder will finish looking into the first patch
[18:36:18] <halfak>	 :) Thanks for taking a look. :) 
[18:44:28] <Helder>	 halfak: patch 1 is now merged
[18:44:33] <halfak>	 Woo!
[18:44:52] * Helder will look the second one
[18:44:57] <halfak>	 I coordinated things so that the second pull request *should* merge without a rebase.
[18:46:37] <Helder>	 halfak: do you know if it is possible to compare only the two latest commits of https://github.com/halfak/Revision-Scoring/pull/20
[18:46:38] <Helder>	 ?
[18:47:14] <Helder>	 i.e. the diff between "Adds a demo for scorer" and "Adds pickle demo to scorer"
[18:50:31] * Helder found it: https://github.com/halfak/Revision-Scoring/compare/master...955923fc5f83c9037965f99e7b2f2bc75fe5a37e
[19:20:59] <Helder>	 halfak: https://github.com/halfak/Revision-Scoring/compare/master...955923fc5f83c9037965f99e7b2f2bc75fe5a37e
[19:21:11] <Helder>	 is that supposed to be test_feature_type?
[19:24:51] <halfak>	 Helder, that link doesn't bring me to a line in particular. 
[19:25:02] <halfak>	 I don't know what you are referring to. 
[19:25:17] <Helder>	 try this:
[19:25:17] <Helder>	 https://github.com/halfak/Revision-Scoring/pull/20/files#diff-0f08f18a0b5c2b30fe9507b7d086885eR21
[19:26:04] <Helder>	 teat_feature_type()
[19:26:05] <halfak>	 Oh.  yes.  typo
[19:26:46] <Helder>	 halfak: in the other pull request I noticed some capitalized "Where"s which could be fixed in the same commit
[19:26:54] * Helder tries to find them
[19:27:40] <Helder>	 halfak: search for "labeled data Where <values_scores> is"
[19:27:48] <Helder>	 on https://github.com/halfak/Revision-Scoring/pull/19/files
[19:27:54] <halfak>	 Looks like that was part of the last pull request. 
[19:28:14] <Helder>	 yes :-)
[19:28:17] <halfak>	 It's fine if you want me to fix them before merging, but it seems unrelated. 
[19:28:19] <danilo_>	 Helder, halfak: I got to use linear kernel reduncing the range and setting probability=False: http://pastebin.com/k5TQ8rhd
[19:28:33] <Helder>	 (it was of minor importance, so I let it go...)
[19:31:25] <Helder>	 halfak: does this date has any special meaning? https://github.com/halfak/Revision-Scoring/compare/master...955923fc5f83c9037965f99e7b2f2bc75fe5a37e#diff-7f54060a935348f181d937a40763b1feR10
[19:31:36] <Helder>	 Timestamp("20050101000000")
[19:32:01] <halfak>	 That's when use registration dates began being recorded in MediaWiki
[19:32:15] <halfak>	 So, it is a conservative lower bound. 
[19:32:30] <halfak>	 Should be documented probably. 
[19:33:12] <Helder>	 got it
[19:36:32] <Ironholds>	 post office trip made!
[19:36:33] <Helder>	 danilo_: I think these values "n / <something>" should probably be computed taking the maximum of the values in the dataset for that feature (instead of hardcoding 10, 100, 50000000, etc..)
[19:36:42] <Ironholds>	 halfak, did you know you can get both Harvey Milk and Boy Scouts of America stamps?
[19:36:55] <Ironholds>	 I bought 20 of each and intend to find excuses to send letters featuring the combination, so I can laugh.
[19:37:05] <halfak>	 Gotta get back to holiday stuff.  Helder, if you give me a list of bits, I'll fix 'em in the pull request next chance I get. 
[19:37:14] <Ironholds>	 enjoy yerself :)
[19:37:27] <halfak>	 Ironholds, this juxtaposition is appreciated :) 
[19:37:31] <halfak>	 Have a good one folks.  
[19:37:32] <halfak>	 o/
[19:37:38] <Helder>	 halfak: you just need to fix the test typo for that pull request
[19:37:48] <Helder>	 halfak: you too!
[19:37:52] <halfak>	 OK.  I'll do that quick and ping. 
[19:41:46] <halfak>	 Helder, got that type in the test. 
[19:41:53] * halfak runs away
[19:41:54] <halfak>	 o/
[19:41:57] <Helder>	 bye :-)
[22:32:37] <halfak>	 Hey Ironholds 
[22:32:42] <halfak>	 Got some more time to hack.  
[22:32:48] <Ironholds>	 cool!
[22:32:49] <halfak>	 I saw your email about hadoop.  
[22:32:58] <halfak>	 I don't know what you are talking about with my job. 
[22:33:05] <halfak>	 I am using less memory and containers than you are right now. 
[22:33:12] * Ironholds checks hadoop again
[22:33:40] <Ironholds>	 yay! This explains why the query ran
[22:34:05] <Ironholds>	 yesterday evening (see the email timestamp) there were two halfak-triggered jobs with ~1TB of memory reserved betwixt them
[22:34:14] <Ironholds>	 I guess one of them must have ended? or both, but replaced by 1 new?
[22:34:43] <halfak>	 I dunno dude.  I've been monitoring these guys.  Also, I haven't *ever* used more than 50 containers for these jobs since I upper-bounded it at that. 
[22:35:05] <halfak>	 The one that is running right now has been running for almost 7 days. 
[22:36:47] <Ironholds>	 yeah, the others were on ~40-50
[22:37:03] <Ironholds>	 But I swear there were two distinct queries yesterday, with approximately 400gb of memory apiece
[22:37:54] <halfak>	 But the hive queries use about the same. 
[23:26:31] <Ironholds>	 stupid google
[23:26:41] <Ironholds>	 I am very, very, very slowly automatically crawling their system
[23:26:52] <Ironholds>	 and by automatically I mean "restarting every 150 queries or so"
[23:32:46] <Ironholds>	 argh.
[23:32:54] <Ironholds>	 identifying duplicated SHAs should not be so hard.
[23:39:06] <Ironholds>	 If eel like this is one of those fundamental CS things with a Knuth algorithm out there somehwere that nobody told me