[11:02:23] <wikibugs>	 06Revision-Scoring-As-A-Service, 06Operations, 06Research-and-Data-Backlog, 10Research-management, and 3 others: [Epic] Deploy Revscoring/ORES service in Prod - https://phabricator.wikimedia.org/T106867#2238896 (10mark)
[13:26:42] <halfak>	 o/
[13:35:08] <Amir1>	 halfak: nlwiki damaging is like russian? (5k edits needing review and needs to be mergred with other 15K?)
[13:47:01] <halfak>	 Good Q.  That might be one of the *new style* 
[13:47:38] <halfak>	 In the new style, we mix 2.5k "needs review" and 2.5k "probably good" so that we can get some observations of "probably good" going wrong. 
[13:47:50] * halfak checks
[13:48:44] <halfak>	 Nope.  Old style.  So yeah.  Mix the 5k needing review with 15 remaining. 
[13:49:30] <halfak>	 It should be about 4160 + 15782
[13:50:12] <Amir1>	 oh, okay
[13:50:15] <Amir1>	 thanks halfak 
[13:50:20] <Amir1>	 going to do it
[13:50:32] <halfak>	 Once we get to one of the new style ones, we'll learn some interesting stuff. 
[13:50:52] * halfak wants to know how many "probably good" edits are damaging. 
[13:50:56] <halfak>	 Or bad faith!
[14:00:00] <halfak>	 We need a good way to convert a TSV into JSON.
[14:00:07] <halfak>	 rather JSON-lines
[14:00:21] * halfak imagines a utility in json2tsv called tsv2json
[14:00:30] <halfak>	 Formatting is going to be a bitch :/
[14:01:47] <halfak>	 Amir1, see https://github.com/wiki-ai/wikilabels/pull/112
[14:01:54] <halfak>	 I just did some updates and merging. 
[14:02:15] <Amir1>	 awesome
[14:02:20] <Amir1>	 thank you halfak 
[14:02:25] <Amir1>	 I have a news
[14:02:44] <Amir1>	 it seems six edits in nlwiki is not labeed
[14:02:49] <Amir1>	 *labeled
[14:03:04] <Amir1>	 and it fails to catch them
[14:03:09] <Amir1>	 what do you suggest?
[14:06:38] <halfak>	 Ignore them
[14:10:19] <Amir1>	 but the fetch_label utility fails
[14:10:21] <Amir1>	 with error
[14:10:36] <Amir1>	 I can make it ignore them 
[14:23:21] <Amir1>	 halfak: ran prelabeling for nlwiki, right now we have 1019 cases of "True" for damaging
[14:23:29] <Amir1>	 Is it expected?
[14:23:38] <Amir1>	 I think so but double check won't make any harm
[14:29:05] <Amir1>	 halfak: I want to close all tasks in done column, is it okay?
[14:50:01] <Amir1>	 halfak: around?
[14:54:02] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: i18n for API errors in wikilabels - https://phabricator.wikimedia.org/T133561#2235853 (10Ladsgroup) https://github.com/wiki-ai/wikilabels/pull/113
[14:55:47] <halfak>	 o/ Amir1 
[14:56:01] <Amir1>	 hey!
[14:56:29] <Amir1>	 halfak: please read above comments
[14:56:45] * halfak looks. 
[14:56:53] <halfak>	 Oh!  So, when you say "prelabel"....
[14:58:21] <halfak>	 Do you mean the "fetch_labels" script?
[14:58:23] <halfak>	 Amir1, ^ 
[14:58:34] <halfak>	 Also, +1 for closing the tasks in the done column. 
[14:58:47] <Amir1>	 no, the prelabel
[14:58:47] <halfak>	 I can do them all in batch if you want. 
[14:58:51] <Amir1>	 since I don't have the dataset
[14:58:53] <halfak>	 Weird.  
[14:58:58] * halfak checks
[14:59:10] <Amir1>	 prelabel is fine
[14:59:22] <Amir1>	 true cases in 5K datasets are rare
[14:59:27] <Amir1>	 (20%)
[14:59:50] <Amir1>	 that is the part that needs your attention
[14:59:53] <halfak>	 Why are you running relabel on a 5k dataset?
[14:59:59] <halfak>	 *prelabel
[15:00:20] <Amir1>	 of course I don't run prelabel on the 5K dataset
[15:00:27] <Amir1>	 I ran it on the 20K one
[15:01:05] <halfak>	 I'm confused.  So when you run prelabel on the 20k sample, you get only 1k that need review?
[15:02:05] <Amir1>	 https://www.irccloud.com/pastebin/kBxlFDta/
[15:03:00] <Amir1>	 halfak: no, when I do grep "True" in nlwiki.rev_damaging.20k_2016.tsv (which is 5K review and 15K other) it returns only 1K
[15:03:12] <halfak>	 Oh!  That's great!
[15:03:23] <halfak>	 We only have like 0.5k damaging in enwiki
[15:03:29] <Amir1>	 ladsgroup@ores-compute-01:~/editquality/datasets$ grep "True" nlwiki.rev_damaging.20k_2016.tsv | wc -l
[15:03:29] <Amir1>	 1019
[15:03:39] <halfak>	 Sorry.  807 in enwiki
[15:03:44] <Amir1>	 oh, okay
[15:03:46] <halfak>	 :)
[15:03:53] <Amir1>	 Ran feature extraction
[15:04:00] <Amir1>	 running tuning reports right now
[15:04:03] <halfak>	 Great!
[15:04:16] <halfak>	 BTW, do you want me to batch resolve the Done column?
[15:04:25] <halfak>	 Or do you have batch rights in phab?
[15:04:40] * Amir1 prays that gradient boosting comes up
[15:04:43] <halfak>	 BRB dog needs attention
[15:04:44] <Amir1>	 I have th right
[15:04:48] <halfak>	 lol @ AMir1's RF models
[15:04:51] <Amir1>	 I do it right now
[15:04:56] <halfak>	 kk back in a bit
[15:05:52] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-articlequality : Train/test `reverted` model for huwiki - https://phabricator.wikimedia.org/T133382#2239707 (10Ladsgroup) 05Open>03Resolved
[15:05:55] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-edittypes, 10wikilabels: Load 200 & 5k samples into wikilabels - https://phabricator.wikimedia.org/T132593#2239708 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup
[15:05:56] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Setup labels-experiment.wmflabs.org - https://phabricator.wikimedia.org/T132588#2239710 (10Ladsgroup) 05Open>03Resolved
[15:05:59] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: DB performance improvements on wikilabels - https://phabricator.wikimedia.org/T132436#2239711 (10Ladsgroup) 05Open>03Resolved
[15:06:01] <wikibugs>	 06Revision-Scoring-As-A-Service, 13Patch-For-Review: Add IRC logging to this board and -backlog - https://phabricator.wikimedia.org/T132359#2239712 (10Ladsgroup) 05Open>03Resolved
[15:06:03] <wikibugs>	 06Revision-Scoring-As-A-Service, 07I18n: Complete the message documentation (qqq) for Revision Scoring - https://phabricator.wikimedia.org/T132208#2239715 (10Ladsgroup) 05Open>03Resolved
[15:06:13] <wikibugs>	 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure, 07Puppet, 03Scap3: deployment-((sca|aqs)01|ores-web) puppet failures due to scap3 errors - https://phabricator.wikimedia.org/T132267#2239713 (10Ladsgroup) 05Open>03Resolved a:03Ladsgroup
[15:06:15] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test `damaging` and `goodfaith` model for ruwiki - https://phabricator.wikimedia.org/T131999#2239718 (10Ladsgroup) 05Open>03Resolved
[15:06:17] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ores, 10rsaas-editquality: Update RTRC to work with new ORES models - https://phabricator.wikimedia.org/T131870#2239719 (10Ladsgroup) 05Open>03Resolved
[15:06:19] <wikibugs>	 06Revision-Scoring-As-A-Service, 10crosswatch, 10ores, 10rsaas-editquality: Update crosswatch to work with new ORES models - https://phabricator.wikimedia.org/T131869#2239720 (10Ladsgroup) 05Open>03Resolved
[15:06:21] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ores, 10rsaas-editquality: Update ScoredRevisions for new models - https://phabricator.wikimedia.org/T131867#2239722 (10Ladsgroup) 05Open>03Resolved
[15:06:23] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ores, 10rsaas-editquality: Update ra-un for new models - https://phabricator.wikimedia.org/T131868#2239721 (10Ladsgroup) 05Open>03Resolved
[15:06:25] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test ruwiki `reverted` model - https://phabricator.wikimedia.org/T131666#2239724 (10Ladsgroup) 05Open>03Resolved
[15:06:27] <wikibugs>	 06Revision-Scoring-As-A-Service, 10revscoring: Language assets for Hindi - https://phabricator.wikimedia.org/T131670#2239723 (10Ladsgroup) 05Open>03Resolved
[15:06:29] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Edit quality campaign for Arabic Wikipedia - https://phabricator.wikimedia.org/T131505#2239725 (10Ladsgroup) 05Open>03Resolved
[15:06:31] <wikibugs>	 06Revision-Scoring-As-A-Service: [Discuss] ORES machine learning as a service for Wikis - https://phabricator.wikimedia.org/T131434#2239728 (10Ladsgroup) 05Open>03Resolved
[15:06:33] <wikibugs>	 06Revision-Scoring-As-A-Service, 10revscoring: Gather language assets for hungarian - https://phabricator.wikimedia.org/T131443#2239727 (10Ladsgroup) 05Open>03Resolved
[15:06:35] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Edit quality campaign for Hungarian Wikipedia - https://phabricator.wikimedia.org/T131446#2239726 (10Ladsgroup) 05Open>03Resolved
[15:06:37] <wikibugs>	 06Revision-Scoring-As-A-Service, 06Research-and-Data: Reach out to CE for Revscoring support - https://phabricator.wikimedia.org/T130893#2239729 (10Ladsgroup) 05Open>03Resolved
[15:06:39] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: [Investigate] Intermittent performance issues with wikilabels - https://phabricator.wikimedia.org/T130872#2239730 (10Ladsgroup) 05Open>03Resolved
[15:06:41] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: i18n wikilabels error messages - https://phabricator.wikimedia.org/T130774#2239733 (10Ladsgroup) 05Open>03Resolved
[15:06:43] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-edittypes, 10wikilabels: Add "no selection made" message to Semantic(s|Operations)Selector form - https://phabricator.wikimedia.org/T130871#2239731 (10Ladsgroup) 05Open>03Resolved
[15:06:45] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-edittypes: Add training campaigns to wikilabels for enwiki and itwiki - https://phabricator.wikimedia.org/T130860#2239732 (10Ladsgroup) 05Open>03Resolved
[15:06:47] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: wikilabels JS should report on server status - https://phabricator.wikimedia.org/T130773#2239734 (10Ladsgroup) 05Open>03Resolved
[15:06:49] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ores: Selective precaching for ORES - https://phabricator.wikimedia.org/T130609#2239735 (10Ladsgroup) 05Open>03Resolved
[15:06:51] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ores: ORES-ui broken - https://phabricator.wikimedia.org/T130536#2239737 (10Ladsgroup) 05Open>03Resolved
[15:06:54] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 13Patch-For-Review: implement an option in preferences to make hidenondamaging default - https://phabricator.wikimedia.org/T130560#2239736 (10Ladsgroup) 05Open>03Resolved
[15:06:56] <wikibugs>	 06Revision-Scoring-As-A-Service: Create puppet role for compute nodes - https://phabricator.wikimedia.org/T130461#2239738 (10Ladsgroup) 05Open>03Resolved
[15:06:59] <wikibugs>	 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: Move ORES extension test MediaWiki install to `ores-staging` project - https://phabricator.wikimedia.org/T130408#2239739 (10Ladsgroup) 05Open>03Resolved
[15:07:01] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels, 07I18n: Message description needed for Wikimedia:Wiki-ai-wikilabels-form-et-ref-label - https://phabricator.wikimedia.org/T130348#2239740 (10Ladsgroup) 05Open>03Resolved
[15:07:03] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for nlwiki - https://phabricator.wikimedia.org/T130290#2239744 (10Ladsgroup)
[15:07:05] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Deploy edit quality models for ruwiki - https://phabricator.wikimedia.org/T130293#2239742 (10Ladsgroup)
[15:07:07] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality, 10wikilabels: Complete ruwiki edit quality campaign - https://phabricator.wikimedia.org/T130270#2239741 (10Ladsgroup) 05Open>03Resolved
[15:07:09] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality, 10wikilabels: Complete nlwiki edit quality campaign - https://phabricator.wikimedia.org/T130267#2239743 (10Ladsgroup) 05Open>03Resolved
[15:07:11] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-edittypes, 10wikilabels: Load 500 revisions for new edit_type campaign pilot - https://phabricator.wikimedia.org/T127081#2239747 (10Ladsgroup) 05Open>03Resolved
[15:07:13] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ores, 13Patch-For-Review, 10Scap3 (Scap3-Adoption-Phase1): Move to using scap3 for deployment for ORES service - https://phabricator.wikimedia.org/T128670#2239745 (10Ladsgroup) 05Open>03Resolved
[15:07:25] <wikibugs>	 06Revision-Scoring-As-A-Service, 10ores: Deploy precaching daemon for ORES - https://phabricator.wikimedia.org/T106638#2239754 (10Ladsgroup) 05Open>03Resolved
[15:07:25] <wikibugs>	 06Revision-Scoring-As-A-Service: Russian language utilities - https://phabricator.wikimedia.org/T106836#2239753 (10Ladsgroup) 05Open>03Resolved
[15:07:28] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Move Wiki Labels javascript to the wiki - https://phabricator.wikimedia.org/T104071#2239756 (10Ladsgroup) 05Open>03Resolved
[15:07:29] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Implement abandoning tasks in Wikilabels - https://phabricator.wikimedia.org/T105521#2239755 (10Ladsgroup) 05Open>03Resolved
[15:07:31] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Log events in the Wiki Labels server - https://phabricator.wikimedia.org/T102347#2239757 (10Ladsgroup) 05Open>03Resolved
[15:12:32] <Amir1>	 "| GradientBoostingClassifier |          0.961 |         0.003 | learning_rate=0.1, n_estimators=300, max_features="log2", max_depth=3 " The giant spaghetti monster answered my prayers, hallelujah 
[15:12:37] <Amir1>	 ramen
[15:14:30] <halfak>	 \o/
[15:14:35] <halfak>	 That's a nice score too
[15:14:51] <halfak>	 We should consider switching to PR-AUC as our scoring metric. 
[15:15:03] <halfak>	 I'll add a phab task for exploring that
[15:16:53] <wikibugs>	 10Revision-Scoring-As-A-Service-Backlog, 10revscoring, 10rsaas-editquality: [Spike] Explore using PR-AUC to score when tuning - https://phabricator.wikimedia.org/T133698#2239792 (10Halfak)
[15:18:57] <halfak>	 https://github.com/wiki-ai/wikilabels/pull/113 looks good.  
[15:19:10] <halfak>	 Amir1, did you test it to make sure there's not some weird syntax error?
[15:19:23] <Amir1>	 yup
[15:19:27] <Amir1>	 I use jshint
[15:19:36] <halfak>	 kk
[15:19:47] <halfak>	 {{done}}
[15:19:47] <AsimovBot>	 How cool, halfak!
[15:19:48] <Amir1>	 (We need to make wikilabels pass jshint too)
[15:21:01] <Amir1>	 thanks halfak :)
[15:21:26] <Amir1>	 halfak: PR-AUC: 0.637 
[15:21:31] <Amir1>	 bad or good?
[15:21:42] <Amir1>	 ROC-AUC: 0.956
[15:22:52] <halfak>	 Really good
[15:23:48] <Amir1>	 I need to change my location, wifi is really bad here
[15:34:10] <wikibugs>	 06Revision-Scoring-As-A-Service, 07I18n: Complete the message documentation (qqq) for Revision Scoring - https://phabricator.wikimedia.org/T132208#2239860 (10Amire80) Thank you!
[15:40:26] <Amir1>	 halfak: I think I made a mistake for goodfaith in Russian
[15:40:35] <Amir1>	 I will fix it with my PR
[15:41:11] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: i18n for API errors in wikilabels - https://phabricator.wikimedia.org/T133561#2239889 (10Ladsgroup) a:03Ladsgroup
[15:41:14] <halfak>	 Sounds good. 
[15:41:19] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test 'damaging' and 'goodfaith' model for nlwiki - https://phabricator.wikimedia.org/T133563#2239890 (10Ladsgroup) a:03Ladsgroup
[16:48:23] <Amir1>	 halfak: nlwiki goodfaith
[16:48:27] <Amir1>	 PR-AUC: 0.998
[16:48:27] <Amir1>	 ROC-AUC: 0.952
[16:48:56] <Amir1>	 too good to be true?
[17:23:01] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Move Wiki Labels javascript to the wiki - https://phabricator.wikimedia.org/T104071#2240395 (10He7d3r) @Ladsgroup, @Halfak: Are we supposed to use https://meta.wikimedia.org/wiki/MediaWiki:Gadget-WikiLabels-loader.js https://meta.wikimedia.org/wiki/MediaWiki:Gadge...
[18:10:34] <halfak>	 Amir1, it looks like that because of the high percentage of positive values. 
[18:10:40] <halfak>	 But that is really good. 
[18:11:06] <Amir1>	 oh great
[18:11:11] <halfak>	 Looks like enwiki :) 
[18:11:23] <halfak>	 See https://ores.wmflabs.org/v2/scores/enwiki/goodfaith/?model_info
[18:11:47] <halfak>	 PR-AUC is 0.997 and ROC-AUC is 0.908
[18:19:07] <Amir1>	 https://github.com/wiki-ai/editquality/pull/29
[18:19:17] <Amir1>	 it seems github bot is down now
[18:20:41] <wikibugs>	 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test 'damaging' and 'goodfaith' model for nlwiki - https://phabricator.wikimedia.org/T133563#2240632 (10Ladsgroup) https://github.com/wiki-ai/editquality/pull/29
[18:21:02] <halfak>	 {{merged}}
[18:21:02] <AsimovBot>	 10[5] 04https://meta.wikimedia.org/wiki/Template:merged
[18:21:07] <halfak>	 \o/
[18:21:21] <Amir1>	 thanks halfak 
[18:21:21] <halfak>	 Kick ass!  We need to do a deployment soon :) 
[18:21:35] <halfak>	 I'm going to try to do that first thing in the morning tomorrow. 
[18:21:38] <Amir1>	 yeah
[18:21:57] <Amir1>	 I will be working on staging in wikilabels
[18:22:04] <halfak>	 great
[18:22:21] <halfak>	 We should do a deployment there soon too.  Do you want to try deploying both ORES and wikilabels to staging?
[18:22:57] <Amir1>	 yeah
[18:23:07] <Amir1>	 that should be straightforward 
[18:23:10] <halfak>	 :) 
[18:23:27] <Amir1>	 btw. I reviewed staging process in wikilabels
[18:23:46] <Amir1>	 As far as I checked everything looks okay except testing it in frontend 
[18:23:56] <Amir1>	 http://labels-staging.wmflabs.org/gadget/loader.js
[18:23:59] <Amir1>	 loader is fine
[18:24:13] <Amir1>	 but how to load it in a test enviroment 
[18:24:28] <Amir1>	 that part is tricky
[19:00:57] <halfak>	 Amir1, looks like I get a server error when loading the gadget
[19:01:00] <halfak>	 http://labels-staging.wmflabs.org/gadget/
[19:01:21] <halfak>	 I have 30 mins, so I'm going to work on getting ORES in staging. 
[19:01:40] <Amir1>	 let me do it
[19:01:53] <halfak>	 kk will do. 
[19:02:03] * halfak looks for other things that can happen in 30 minutes. 
[19:04:09] <Amir1>	 regarding the wikilabels issue
[19:04:25] <Amir1>	 it seems I have ssh connection issues
[19:05:40] <Amir1>	 no, I'm dumb
[19:06:19] <Amir1>	 "no python application found, check your startup logs for errors"
[19:06:30] <Amir1>	 okay. Fixing it should be simpel
[19:06:33] <Amir1>	 *simple
[20:23:22] <halfak>	 Amir1, just about to take off to run some errands. 
[20:23:29] <halfak>	 How's ORES-->Staging going?
[20:34:09] * halfak --> away for a few hours
[20:45:32] <Amir1>	 halAFK: hey, please check telegram
[20:46:03] <Amir1>	 I was afk for some offline work
[23:01:58] <wikibugs>	 06Revision-Scoring-As-A-Service, 10wikilabels: Review staging protocol for Wikilabels - https://phabricator.wikimedia.org/T133557#2241625 (10Ladsgroup) Here's my review.   - puppet      - There is no issue with puppet, it's generalized enough      - Since both wikilabels and wikilabels-staging coexist in a pro...