[15:49:41] o/ [15:50:56] o/ halfak [15:52:06] Not feeling great this morning. We'll see how far I get on KDD methods and the ORES UI deploy [15:53:48] okay :) [15:53:54] I would be happy to do what I can [15:58:39] right now I'm writing a script for fa.wp [15:58:46] it'll be done in ten min [16:08:45] Amir1, I wanted to talk to you about a problem that I found. [16:09:07] sure harej [16:09:11] sorry wrong ping [16:09:15] sure halfak [16:09:21] * Amir1 curses at circ [16:09:38] When I give Random Forest or GradientBoosting the fawiki damaging dataset, the threshold for a true prediction is so high that it doesn't predict true in the test set. [16:09:50] I think this is because the dataset has so few true obs. [16:10:52] I see [16:11:08] We actually planned to have another campaign for fa.wp [16:11:18] but I don't know why we didn't do it [16:12:10] So, what I'd suggest for the campaign is to pre-process another 20k obs for non-trusted edits. That should get us another 500 or so True obs. [16:12:23] And 1-2k will need to be labeled. [16:13:21] I can ask users to do that [16:13:27] It won't be a big problem [16:13:45] +1 then :) [16:13:50] do you think one another 20K is enough? [16:14:15] Amir1, if not, we can do a bigger sample and prelabel. [16:15:53] I think we do it again [16:16:02] if it wasn't enough, let's do another round [16:21:08] +1 [16:21:24] OK. I have a set of papers and I'm digging through them and taking notes. :) [16:22:37] for KDD? [16:24:10] Yup [16:26:01] awesome [16:26:09] I can send you mine [16:27:17] :) [16:35:00] halfak: During my lit. review I sent my notes to Arthur and you through email [16:35:10] that's where PCFG came from [16:35:17] I think you should take a look at them [16:38:58] Subject? [16:44:44] Let me search [16:46:10] "Three more interesting papers" [16:46:14] Thanks [16:46:25] "bag of words approach" [16:46:46] "some notes" [16:47:07] Great. [16:47:23] I'll likely only get through a few of these though. [16:47:30] Did you add these all to your refs.bib? [16:47:42] I think so [16:47:48] I'm working on the set I could find in the PDF of the draft [16:47:49] Thanks to google scholar [16:48:56] Basically almost all of papers are included in the refs [16:49:04] of draft pdf [16:51:34] Perfect. That's what I'm working from now. [16:54:06] halfak: what Should I do for now? [16:54:12] Add more? [16:55:07] Re. paper, I think we need an essay on why realtime (aka zero-delay) features and feature-computation is the only model that really matters for Wikipedians. [16:55:55] It seems that all of the papers primarily (if not only) talk about post-hoc (aka historical) features or features that are nearly impossible to compute in realtime. [16:56:24] Props to A West and Adler for talking about zero-delay and making the use-case distinction. [16:57:02] Yeah I see [16:57:08] What do you think? [16:57:12] Sound interesting? [16:57:40] one of papers (which didn't get any attention) was computing edits based on search on google and checking first 150 results [16:57:50] = complete waste of resources [16:57:54] Yeah... That. [16:58:20] Write using whatever voice/style you want and we can adapt to research paper voice/style later. [16:58:43] okay :) [16:59:41] starting right now [17:02:16] halfak: an update regarding the extension, we are getting some helps from Wikidata people \o/ [17:02:20] in code review [17:14:10] halfak: shared [17:14:32] it's just one rather big paragraph [17:44:40] Amir1, we might want to release a PAN-like dataset for Wikidata. [17:44:51] It seems that this will help people build off of our work. [17:45:03] +1 [17:45:33] Also, one more thing. I think a big part of the counter-intuitive nature of our classifier output is because we train it with a balanced dataset [17:45:56] If we unbalance the dataset, we'll get better probability estimates. [17:46:34] We might also be able to use "weights" in sklearn. [17:47:26] I like to see a trial of unbalanced [17:47:33] and see what happens [17:47:34] +1 [17:49:04] We fool our SVC's into thinking that they get balanced, but we don't need to fool the random forest or gradient boost. [17:49:19] * halfak makes a note that we should add adaboost to our tuning. [18:03:47] Some of these papers fail to make sure that, when they reference a stat, they grab the stat that came from the *test* set. I'm seeing some *training* set stats reported as though they represent the fitness. >:( [18:04:03] I like when both are reported, but never *just* report stats on your training set. [18:07:10] Amir1, https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service/Work_log/2016-01-23 [18:07:22] That's as far as I got. I should look at merging the ORES UI and getting it deployed. :) [18:07:50] awesome [18:07:51] thanks [18:08:56] I think we should plot the distribution of score-timings for our model. I.e. how long one must wait before ORES can generate a score. [18:09:15] This will account for CPU/IO involved in getting data, extracting features and applying the model. [18:09:28] And make a call for future work to do the same. [18:09:52] aweomse [18:10:00] I'm reading them [18:10:05] this is great [18:10:23] I'll do other papers [18:10:43] the way you've done the first ones [18:13:17] Great :) I'm guessing this will turn into a nice to-do list for us. [18:13:47] E.g. Make a feature table, build models using subsets of features and report, generate score-speed distribution graph. [18:16:50] yes, [18:17:04] and we are trying to get the vagrant patch merged [18:17:10] (in the mean time) [18:19:05] {{merged}} UI [18:19:05] 10[1] 04https://meta.wikimedia.org/wiki/Template:merged [18:22:38] \o/ [18:23:02] When we should get this deployed? [18:23:43] GOing to check it out on staging now [18:27:31] the vagrant is merged now :) [18:28:13] Great! [18:28:26] Hmm... looks like something weird is up with our staging server [18:28:29] YuviPanda, around? [18:28:43] On ores-staging, I'm getting "uwsgi-ores-web: unrecognized service" [18:28:53] It seems that the service named "uwsgi" is running ORES. [18:29:28] This happened once before when I manually shut down uwsgi-ores-web and started uwsgi manually. I definitely didn't do that this time. [18:29:38] The recent reboot might have broke something? [18:31:21] I doubt that [18:31:26] but let's see [18:31:33] Amir1, http://ores-staging.wmflabs.org/ui/ [18:32:05] it gave me 404 when I tried it ~10 min ago [18:32:11] it works [18:32:14] \o/ [18:32:14] :) [18:32:58] OK, the university is closing, I got to go home and continue from there :) [18:33:06] it's 10 PM here [18:33:12] see you in 20 min. [18:34:57] o/ [18:35:25] I might be gone when you get back. Starting to feel sick again. Had food poisoning (I think) last night. :( [18:35:57] Yup. Looks like the uwsgi-ores-web is missing on our "prod" web nodes too. [18:36:09] I want to have YuviPanda comment before I do a real deploy, I think [19:12:52] I hope you feel better soon halfak [19:13:10] my connection is bad right now [19:13:20] let me find an alternative [19:29:23] violetto: hey, the GUI is deployed in our staging server, ores-staging.wmflabs.org/ui/ [19:29:39] one thing: It seems we have a bug with firefox, [19:29:48] can you check that and suggest? [20:21:57] halfak: if you're around: https://meta.wikimedia.org/wiki/Research_talk:Revision_scoring_as_a_service/Work_log/2016-01-23#harpalani11language [20:30:19] hey halfak [20:30:21] checking [20:32:30] halfak: I see what you mean and it was definitely caused by the restart [20:34:30] halfak: sudo service uwsgi stop and sudo service uwsgi-ores-web start should fix that [20:34:36] and then we should file a bug and fix it in puppet [21:06:00] (03PS1) 10Ladsgroup: Some minor improvements to the database schema [extensions/ORES] - 10https://gerrit.wikimedia.org/r/265944 (https://phabricator.wikimedia.org/T124443) [21:11:28] I've done that on the two prod machine [21:11:31] s [21:18:25] (03PS2) 10Ladsgroup: Some minor improvements to the database schema [extensions/ORES] - 10https://gerrit.wikimedia.org/r/265944 (https://phabricator.wikimedia.org/T124443) [23:27:01] Amir1: woop!