[10:25:29] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: [research] Why is the japanese 'reverted' model so bad? - https://phabricator.wikimedia.org/T133405#2314830 (10Ladsgroup) a:03Ladsgroup [10:47:22] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: [research] Why is the japanese 'reverted' model so bad? - https://phabricator.wikimedia.org/T133405#2314840 (10Ladsgroup) Here's my first results. - Number of reverted edits are super small (382) which means we are prone to overfitting. Let me increas... [13:04:57] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: [research] Why is the japanese 'reverted' model so bad? - https://phabricator.wikimedia.org/T133405#2314960 (10Ladsgroup) With 40K edits: ``` ScikitLearnClassifier - type: GradientBoosting - params: balanced_sample=false, max_leaf_nodes=null, center=true... [13:09:52] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: [research] Why is the japanese 'reverted' model so bad? - https://phabricator.wikimedia.org/T133405#2314961 (10Ladsgroup) Another thing I learned from Japanese Wikipedia is that number of unregistered users doing good edit is much more than other wikis. Jus... [14:10:44] o/ [14:19:12] halfak: o/ [14:21:11] halfak: I ran some checks on https://phabricator.wikimedia.org/T133405 [14:22:13] Wow. Still, I would expect 0.85ish ROC-AUC range when we drop protected user features. [14:22:41] Could you get a random sample of reverted edits and non-reverted edits for a local jawiki user to review for us? [14:23:04] If our train/test set is high quality then I think we should just deploy this classifier. But if it is low quality, we should re-examine what's going on. [14:23:22] halfak: sure [14:24:53] halfak: I also checked PR-AUC for Wikidata, it gives virtually the exact same results as ROC-AUC [14:25:01] not the number, order [14:28:25] Gotcha. OK. I guess that makes sense. [14:29:31] BTW, we could maybe even publish a paper about that since there's been substantial discussion about using PR-AUC rather than ROC-AUC. [14:29:41] It would be short paper for sure. [14:30:23] you're the scientist guy of the group :D [14:30:47] :) [14:30:49] halfak: I will check some other languages too [14:31:00] I have trouble with English [14:31:07] tuning report doesn't work at all [14:34:39] Weird. [14:34:44] Does it error out? [14:39:30] yup [14:39:52] What kind of error? [14:39:57] halfak: 1- It adds lines like "Cache-id: 000" [14:40:06] 2- and then every report is zero [14:40:09] Yeah. Sorry. Pull the newest version of revscoring. [14:40:19] let me try again [14:40:20] ok [14:40:28] I discovered that while hacking on the plane and released a new version. [15:05:56] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: [Spike] Proof of concept damage detection with hash vectors - https://phabricator.wikimedia.org/T132581#2315087 (10Halfak) **Stacking**: You know how you built a model just based on the vectorizer features first. Use that models predict_proba(...)... [15:33:51] halfak: around? [15:34:07] two things: 1- I deployed wikilabels a couple days ago [15:34:48] Also I reverted a patch I made wrt good connection check [15:35:02] since it got tons of errors in staging [15:35:30] but now I see it's not related and our staging setup has other issues connecting to the database, always get a bad connection [15:35:43] so I will re-revert and deploy very soon [15:36:58] 2- I merged my own PR about changing name of 'uwsgi-ores-web' to 'uwsgi-ores'. TLDR: We can't deploy unless the Alex's patch gets merged (and once it's merged, I need to switch off old services manually, so we would have some down time) [15:37:13] which will happen very soon [16:08:58] Amir1: you don't need to ahve downtime [16:09:09] Amir1: you can just turn it off in one instance first, let other handle the load, bring it up, do again [16:16:14] halfak: Amir1 http://queue.acm.org/detail.cfm?id=2841313 was a very nice read wrt automating systems [16:25:39] I'm reading about research on automation as I attempt to automate jupyterhub deploys [18:44:02] YuviPanda: Hey, I was afk, sorry [18:44:23] I'm not sure we do have a proper subscription system for our lb [18:44:44] it will work for 'connection refused' [18:45:59] uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html [18:46:20] hmm [18:46:24] on production we use something totally different (LVS) [18:46:35] YuviPanda: It would be great [18:46:50] we can't use it in labs tho [18:47:20] yeah, I'm not sure if we need to do anything special for LVS [18:47:24] I checked some stuff