[08:58:23] hey awight [08:59:23] just wanted to do a follow-up to last week's meeting [09:00:02] posting all my scripts and plots here: https://github.com/saurabhbatra96/wmf-samplecodes [09:02:09] i did get the PR curves for an SVM, although they don't look promising, going to try out random forest next [11:29:28] awight: ping! me and Amir1 have a question - do you know why ores vm in deployment-prep is shut down? [11:29:42] I need it to test something, was wondering what's up [11:30:13] awight: If you got back and feel reviewing this is for you: https://gerrit.wikimedia.org/r/#/c/434689/ [11:35:42] Pchelolo: that's bad, it shouldn't be down. [11:36:08] Amir1: I'll try to get to that later tonight... [11:36:25] Thanks! [11:36:29] from horizon it seems it's shutoff.. but I don't think I have permissions to try to restart the vm [11:39:26] Fwiw, the last thing I did with it was to fix puppet [11:39:30] hi awight [11:39:43] Someone stopped it this morning, 8:42 utc [11:39:52] saurabh: hi! Just taking a look at yr repo now [11:39:59] cool! [11:40:38] wanted to ask what other plots and classifiers should I try out? [11:40:54] did start some work with random forests [11:42:19] gaussian boost maybe... I don't have experience in choosing these, but just know it's been working well for our completely different problem domain :) [11:42:41] alright, did not do the PR plot for random forests [11:42:57] but was getting an average accuracy score in the lower 90s which seemed promising [11:43:37] i'll try random forests and gaussian boost, then maybe we can meet up and compare them [11:43:54] Pchelolo: Amir1: I restarted http://ores-beta.wmflabs.org/ and asked -releng who might have stopped it [11:47:10] saurabh: Amir1 knows much more than me about ML, btw :-) if anything from https://github.com/saurabhbatra96/wmf-samplecodes/ jumps out at him [11:48:02] haven't gotten around to writing any documentation, so might be tough to follow without context [11:49:03] but we're basically doing binary classification on a balanced dataset of size ~1k entries [11:52:55] awesome, thank you awight [11:55:13] :) [11:55:23] Glad someone noticed ores-beta was dead! [11:55:50] saurabh: hey, that sounds nice. How I can help you? [11:57:11] just wanted to ask what kind of things do you usually look at when deciding which ML model to opt for [11:57:23] for ex. currently I'm generating PR curves [11:57:35] https://github.com/saurabhbatra96/wmf-samplecodes/tree/master/plots [12:02:12] saurabh: It might be fun to write these using Jupyter, btw [12:04:51] saurabh: cv_train might give a slight fitness boost, since you can train using the entire set [12:05:04] (so 50% more data) [12:05:52] never heard of it, do you have a reference handy? [12:06:19] http://scikit-learn.org/stable/modules/cross_validation.html [12:09:13] ohh, cross-validation. thanks, this does feel promising [12:09:28] i'll update the SVM code with it and see if it improved [12:10:12] saurabh: it's also worth doing hyper-parameter tuning before dismissing algorithms, of course: http://scikit-learn.org/stable/modules/grid_search.html [12:12:20] yup, makes sense. I remember you sent a link to a table documenting accuracy scores vs hyperparameters [12:12:29] for some other WMF AI project [12:12:50] I guess I'll do the same, spend some more time on getting info about a model [12:13:00] Again, I wish I had the experience to recommend anything about tuning, but I don't yet :-/ [12:14:01] I'll do some reading up, there's going to be a lot of material out there I'm sure [12:15:41] Feel free to send anything useful my way so I can keep up! [12:17:26] Yup, I'll send you a mail accross. Expect it by Friday [12:18:41] which isn't exactly the best day, but we can discuss next monday if friday won't work [12:21:04] saurabh: You've seen this page, right? https://stripe.com/radar/guide [12:21:09] pretty awesome level of detail. [12:21:45] In this section they say that RF is what they use for most tasks, https://stripe.com/radar/guide#how-does-machine-learning-work [12:23:13] yup I did go through it once [12:23:39] This jumped out... "Though we constantly iterate on and experiment with our modeling process at Stripe, we’ve found that random forests (a generalization of decision trees like the one above) work well for a wide swath of the machine learning problems we face." [12:24:06] that corroborates well with the results I have until now [12:24:19] accuracy scores in the 90s are pretty much a dream come true [12:26:12] Looking forward to seeing more statistics for RF! [12:26:54] i'll try and do comprehensive analysis for svm, rf and hopefully logistic regression if I can [12:27:00] by end of this week [12:28:26] Go slower :) [12:28:36] end of this weekend? :-) [12:28:39] hehe [12:28:49] Have a look at this [12:28:51] https://stripe.com/radar/guide#break-even-calculator [12:29:22] Just what we were discussing the other day. How do we decide what tradeoff works b/w P and R [12:29:23] yes totally [12:29:55] then there's the human element too, MBeattie will have some ideas about that [12:31:19] signing off for now, will drop you a mail soon [12:31:21] :-) [12:31:23] thanks! [12:31:46] o/ [12:51:56] (03CR) 10Awight: "questions" (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/434689 (https://phabricator.wikimedia.org/T194297) (owner: 10Ladsgroup) [13:03:02] halfak: Apparently I have to ask to join our meeting this morning, I'm trying to connect, if someone could let me join that would be great. Thanks! [13:03:19] I'm officially AFK, but let me see if I could help! [13:03:32] Ah, sorry! [13:04:00] Amir1! MEeeting! [13:04:15] Can't find a place [13:04:18] :S [13:47:32] 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, 10EventBus, and 4 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#4242922 (10Ottomata) [13:47:38] 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, 10EventBus, and 4 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#3319960 (10Ottomata) [13:48:17] 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, 10EventBus, and 4 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#3319960 (10Ottomata) a:03Ottomata [13:51:31] 10Scoring-platform-team (Current), 10editquality-modeling, 10Patch-For-Review, 10User-Ladsgroup, 10artificial-intelligence: Train damaging/goodfaith models for Bosnian Wikipedia - https://phabricator.wikimedia.org/T194876#4242930 (10Ladsgroup) https://github.com/wiki-ai/editquality/pull/161 [13:56:53] (03CR) 10Ladsgroup: Make aggregation of score results possible so they take less rows (032 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/434689 (https://phabricator.wikimedia.org/T194297) (owner: 10Ladsgroup) [14:07:13] 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, 10EventBus, and 4 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#4242999 (10Pchelolo) [15:08:42] 10Scoring-platform-team, 10Analytics, 10EventBus, 10ORES, 10User-Ladsgroup: Numeric keys in ORES models causing downstream Hive ingestion to fail - https://phabricator.wikimedia.org/T195979#4243147 (10Ottomata) [16:54:06] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10artificial-intelligence: arwiki goodfaith model is not usable - https://phabricator.wikimedia.org/T193905#4243387 (10Ladsgroup) Okay, I have been checking this, I know a little bit of Arabic (long story). First [[http://labels.wmflabs.org/stats/arw... [17:38:10] awight: hey, tell me when you have some time I have lots of things for you to review :D [17:58:41] Amir1: right on [17:59:04] I can do some stuff in a few hours when the wild monkeys are asleep [17:59:17] but feel free to link any time, of course [17:59:38] :))) [17:59:59] I'm deploying at the moment [18:00:04] (SWAT) [18:06:46] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10artificial-intelligence: Enable ORES filters on srwiki - https://phabricator.wikimedia.org/T195870#4243642 (10jmatazzoni) [18:07:20] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10artificial-intelligence: Enable ORES filters on srwiki - https://phabricator.wikimedia.org/T195870#4239683 (10jmatazzoni) a:03Catrope [18:07:42] awight: https://github.com/adamwight/ores-reference-ui/pull/1 https://github.com/wiki-ai/editquality/pull/161 https://github.com/wiki-ai/wikilabels/pull/237 https://gerrit.wikimedia.org/r/#/c/434689/ [19:15:36] Amir1: oops, I should have mentioned that I moved active work over to the wiki-ai/ores-reference-ui repo [19:18:22] awight: you forked it which caused the github to confuse and make this PR against your repo, it would be great if you delete your repo [21:03:39] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/436387 (owner: 10L10n-bot) [21:39:31] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10artificial-intelligence: arwiki goodfaith model is not usable - https://phabricator.wikimedia.org/T193905#4244454 (10alanajjar) Hello all, Is there any help that I can do here?