[00:15:41] Universal adversarial perturbations https://arxiv.org/abs/1610.08401 seems to be a huge finding in ML that renders images unclassifiable [00:15:56] if anyone is interested in some "light" reading :) [00:23:09] interesting. Thanks bearloga [00:24:15] Cool! [00:25:05] I wonder if this is a natural characteristic of the distributed nature of a network model. Or maybe it's a more fundamental property of pattern detection in bitmaps. [00:25:23] quasi-relevant HN discussion of paper: https://news.ycombinator.com/item?id=12824380 [00:53:05] 10Revision-Scoring-As-A-Service-Backlog, 10revscoring: `revscoring tune` should do basic reporting to stderr - https://phabricator.wikimedia.org/T161582#3135999 (10Halfak) [02:39:40] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), and 2 others: Enable parts of ORES extension by default and manage impacts on the RC Page and the RC page Preferences tab - https://phabricator.wikimedia.org/T159763#3136107... [08:52:51] 06Revision-Scoring-As-A-Service, 10ORES, 10Wikimedia-Logstash, 13Patch-For-Review, 15User-Ladsgroup: Send ORES logs to logstash - https://phabricator.wikimedia.org/T149010#2739588 (10Ladsgroup) From logstahs: [pid: 28235] 10.2.2.10 (-) {32 vars in 699 bytes} [Tue Mar 28 08:50:44 2017] GET /v2/scores/enwi... [08:53:12] 06Revision-Scoring-As-A-Service, 10ORES, 10Wikimedia-Logstash: Send ORES logs to logstash - https://phabricator.wikimedia.org/T149010#3136397 (10Ladsgroup) 05Open>03Resolved p:05Triage>03Normal a:03akosiaris [08:55:27] 06Revision-Scoring-As-A-Service, 10ORES, 10Wikimedia-Logstash: Send ORES logs to logstash - https://phabricator.wikimedia.org/T149010#3136402 (10akosiaris) Heh, you beat me to it :-) I 've also created https://logstash.wikimedia.org/app/kibana#/dashboard/ORES and added a link from the home page. I am wonde... [13:18:27] o/ [13:56:24] o/ Amir1 [13:56:37] I'm running late. Will PM. Something came up last night. [13:56:43] halfak: sure [13:56:53] The meeting just finished and I'm around [14:13:34] Amir1, I'm extracting features for kowiki now. [14:13:41] I'll have some updates to wheels shortly. [14:13:46] nice [14:30:46] Amir1, https://gerrit.wikimedia.org/r/345150 [14:31:15] * halfak cross-validates the kowiki reverted model [14:31:19] Almost there... [14:31:25] {{merged}} [14:31:26] 10[1] 04https://meta.wikimedia.org/wiki/Template:merged [14:37:32] I'm making changed to the deploy repos in anticipation [14:48:03] I'm reading some articles about LSTM, ping me if anything pops up. I will also update the phab card for initial assessment of TF [14:48:14] halfak: (forgot to ping) [14:48:16] OK will do. [14:50:00] http://colah.github.io/posts/2015-08-Understanding-LSTMs/ [14:50:03] http://karpathy.github.io/2015/05/21/rnn-effectiveness/ [14:50:15] these are two really interesting [15:06:51] Amir1, https://github.com/wiki-ai/editquality/pull/62 [15:07:01] Once that is merged, I think I'll be ready for the deploy repos. [15:07:28] * Amir1 reads [15:08:56] +1 for your note in the Makefile. [15:09:08] I plan to clean that all up some day. We should have original query links. [15:09:53] halfak: hewiki damaging is RF model [15:10:12] they add a lot to memory footprint [15:10:20] maybe we should reconsider :D [15:10:21] Right. But the boost is substantial. [15:10:37] yeah, 1.3% which is a lot in this context [15:11:18] yeah, let's have it for now and be in sync with Ops when deploying it [15:11:52] merged [15:12:06] COol [15:12:58] The wikipedia part in this LSTM article is really fun: http://karpathy.github.io/2015/05/21/rnn-effectiveness/ [15:13:23] they trained it on Wikipedia article and was able to produce fake, article-sounding stuff [15:15:27] https://github.com/wiki-ai/ores-wmflabs-deploy/pull/78 [15:17:04] quick question halfak. My mind is not helping but do we need to update anything on ores repo when adding a new wiki? [15:17:10] I doubt that but just double checking [15:18:01] nope [15:18:18] Most important bits are revscoring version (if new language) and relevant dependencies. [15:18:36] In this case, no new enchant dictionaries, but we did have a bump in deltas for handling ko better. [15:19:07] okay, merged :) [15:20:10] Woops. I just pushed a remote branch out to our prod repo and that worked. [15:22:13] goddamn gerrit won't accept my changes now. [15:22:16] arg! [15:26:51] Amir1, https://gerrit.wikimedia.org/r/345162 [15:26:55] That was so hacky [15:27:19] I added a period to the end of our footer sentence to get a new change ID [15:28:11] 06Revision-Scoring-As-A-Service, 10Bad-Words-Detection-System, 10revscoring: Add language support for Korean - https://phabricator.wikimedia.org/T160757#3109896 (10Halfak) [15:28:18] 06Revision-Scoring-As-A-Service, 10Bad-Words-Detection-System, 10revscoring: Add language support for Korean - https://phabricator.wikimedia.org/T160757#3109896 (10Halfak) a:03Halfak [15:28:23] it surprised me a little to see unrelated change but I though meh [15:28:24] 06Revision-Scoring-As-A-Service, 10Bad-Words-Detection-System, 10revscoring: Add language support for Korean - https://phabricator.wikimedia.org/T160757#3109896 (10Halfak) https://github.com/wiki-ai/revscoring/pull/306 [15:28:25] :D [15:28:53] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test reverted model for kowiki - https://phabricator.wikimedia.org/T161616#3137234 (10Halfak) [15:29:04] 06Revision-Scoring-As-A-Service, 10Bad-Words-Detection-System, 10revscoring: Add language support for Korean - https://phabricator.wikimedia.org/T160757#3137250 (10Halfak) [15:29:06] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test reverted model for kowiki - https://phabricator.wikimedia.org/T161616#3137249 (10Halfak) [15:29:37] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test reverted model for kowiki - https://phabricator.wikimedia.org/T161616#3137234 (10Halfak) https://github.com/wiki-ai/editquality/pull/62 [15:30:04] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train/test reverted model for kowiki - https://phabricator.wikimedia.org/T161616#3137254 (10Halfak) [15:30:06] 06Revision-Scoring-As-A-Service: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3137253 (10Halfak) [15:30:10] heh [15:30:14] halfak: Do you want to deploy for prod today? [15:30:18] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3106151 (10Halfak) [15:30:23] I think so. [15:30:31] But I'm not in that much of a rush. Tomorrow would be OK too. [15:30:47] I think we should stay in labs and beta for just a day [15:31:04] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train damaging and goodfaith models for hewiki - https://phabricator.wikimedia.org/T160930#3115376 (10Halfak) https://github.com/wiki-ai/editquality/pull/62 [15:31:15] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Train damaging and goodfaith models for hewiki - https://phabricator.wikimedia.org/T160930#3137258 (10Halfak) a:03Halfak [15:31:33] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Deploy edit quality models for hewiki - https://phabricator.wikimedia.org/T130284#3137259 (10Halfak) [15:31:56] 06Revision-Scoring-As-A-Service, 10rsaas-editquality: Deploy edit quality models for hewiki - https://phabricator.wikimedia.org/T130284#2131647 (10Halfak) [15:31:58] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3106151 (10Halfak) [15:32:07] Amir1, sounds good to me. [15:32:23] I can do the deployment if you're afk at that time [15:32:50] 06Revision-Scoring-As-A-Service, 10ORES, 10rsaas-editquality: Deploy reverted model for kowiki - https://phabricator.wikimedia.org/T161617#3137264 (10Halfak) [15:32:56] 06Revision-Scoring-As-A-Service, 10ORES, 10rsaas-editquality: Deploy edit quality models for hewiki - https://phabricator.wikimedia.org/T130284#2131647 (10Halfak) [15:34:00] 06Revision-Scoring-As-A-Service, 10ORES, 10rsaas-editquality: Deploy reverted model for kowiki - https://phabricator.wikimedia.org/T161617#3137289 (10Halfak) https://github.com/wiki-ai/ores-wmflabs-deploy/pull/78 [15:34:09] 06Revision-Scoring-As-A-Service, 10ORES, 10rsaas-editquality: Deploy edit quality models for hewiki - https://phabricator.wikimedia.org/T130284#3137290 (10Halfak) https://github.com/wiki-ai/ores-wmflabs-deploy/pull/78 [15:34:20] 06Revision-Scoring-As-A-Service, 10ORES, 10rsaas-editquality: Deploy edit quality models for hewiki - https://phabricator.wikimedia.org/T130284#3137291 (10Halfak) https://gerrit.wikimedia.org/r/345162 [15:34:33] 06Revision-Scoring-As-A-Service, 10ORES, 10rsaas-editquality: Deploy reverted model for kowiki - https://phabricator.wikimedia.org/T161617#3137292 (10Halfak) https://gerrit.wikimedia.org/r/345162 [15:37:35] just wondering... https://github.com/wiki-ai/ores-wmflabs-deploy/pull/78/commits/2fe844223401d08b22250fd1e910c221a9a88db0#diff-77ae19360cfaa1aaaea7086fab249bf4R395 not alphabetical? [15:37:39] I'm going to do a deploy to wmflabs and beta now. [15:38:00] revi, what's not alphabetical? [15:38:13] Oh! the api list [15:38:20] yeah. Looks like that got mixed up. [15:38:23] yeah [15:38:46] too trivial to make a PR tho [15:38:51] so fix it when you remember :-p [15:41:57] I'm busy IRL for this week, so I won't be much available [15:42:18] if something needs to be done from my side, ping me and I'll do it next week :-p [15:43:27] revi, mostly the same for me. Will likely pick this up again next week. [15:43:40] Once we get the deploy done, the ScoredRevisions gadget will work for kowiki [15:43:56] We'll need to do a bit more work to get the ORES extension working. [15:44:10] gadget is probably my side working [15:44:12] :-p [15:44:24] ? [15:44:29] do we have the campaign up? [15:44:34] Amm [15:44:36] hmm [15:44:36] Amir1, not yet. We have autolabels. [15:44:37] * [15:44:44] I thought it was gadget in wiki? [15:44:49] okay [15:44:54] revi, you can install the gadget from Meta. [15:45:06] yeah, it requires sysop [15:45:11] revi, https://github.com/he7d3r/mw-gadget-ScoredRevisions [15:45:12] nope [15:45:20] making it gadget in prefa [15:45:22] prefs* [15:45:24] It does if you want the check box [15:45:25] yeah [15:45:32] But you can use it like this anyway. [15:45:36] Yeah [15:45:43] I just install via meta's gloabl.js and then have it everywhere :) [15:45:56] https://meta.wikimedia.org/wiki/Special:MyPage/global.js [15:46:05] revi: I don't recom. that because that's temp. and the beta feature should replace it [15:46:17] (I'm talking about making a gadget in prefs) [15:46:26] we did that in fa.wp and it wasn't fun [15:46:36] Amir1, we should have a conditional in ScoredRevisions that disables it on Change lists when the beta feature is running. [15:46:59] Though to be honest, it does play nicely. [15:47:05] yeah, it's something we need to do soon [15:47:19] but puts lots of pressure on the server [15:47:23] :D [15:47:36] Meh. Cached scores :) [15:47:46] If the ORES beta feature has 'em, redis has 'em too. [15:47:50] Think of th epolar bears [15:47:57] *the polar [15:49:28] ORES is now enabled by default in ptwiki! [15:49:40] yup [15:49:46] pt and pl [15:49:49] Nice [15:50:05] https://ores-staging.wmflabs.org/v2/scores/hewiki [15:50:17] https://ores-staging.wmflabs.org/v2/scores/kowiki [15:50:30] https://ores-staging.wmflabs.org/v2/scores/kowiki/reverted/2456377 [15:50:33] revi, ^ [15:51:20] heh [15:51:22] thanks [15:51:58] Amir1, do you know a good way to check if ORES is enabled via javascript? [15:52:09] *Ores review tool [15:52:18] yeah, there are some config vars [15:52:25] mw.config.get('foo') [15:52:36] check it on English Wikipedia [15:52:40] kk will do [15:53:30] mw.config.get('oresThresholds') [15:53:33] "oresData" [15:53:37] Works for me. [15:53:38] halfak: one of them [15:53:42] this is smaller [15:53:50] oresData is a much bigger chunk [16:06:43] http://ores-beta.wmflabs.org/v2/scores/hewiki [16:06:51] http://ores-beta.wmflabs.org/v2/scores/kowiki [16:06:57] Looks like we're ready to go [16:10:51] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3137407 (10Halfak) This is now live at https://ores.wmflabs.org and https://ores-beta.wmflabs.org. We'll be doing a deployment for https://ores.wikimedia.org tomorrow. [16:12:15] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Deploy ORES Review Tool for hewiki - https://phabricator.wikimedia.org/T161621#3137415 (10Halfak) [16:12:24] 06Revision-Scoring-As-A-Service, 10ORES: Deploy ORES late march - https://phabricator.wikimedia.org/T160638#3106151 (10Halfak) [16:12:27] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 15User-Ladsgroup: Deploy ORES Review Tool for hewiki - https://phabricator.wikimedia.org/T161621#3137432 (10Halfak) [16:13:25] 06Revision-Scoring-As-A-Service, 10Wikilabels, 10rsaas-editquality: Deploy editquality campaign in Korean Wikipedia - https://phabricator.wikimedia.org/T161622#3137435 (10Halfak) [16:13:43] revi, can you translate "Edit quality (20k sample)" into Korean for me? [16:14:08] 편집 품질 (예시 2만 개) [16:14:29] Thanks! [16:32:24] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), and 2 others: Enable parts of ORES extension by default and manage impacts on the RC Page and the RC page Preferences tab - https://phabricator.wikimedia.org/T159763#3137565... [16:35:14] 06Revision-Scoring-As-A-Service, 10Wikilabels, 10rsaas-editquality: Deploy editquality campaign in Korean Wikipedia - https://phabricator.wikimedia.org/T161622#3137572 (10Halfak) ``` halfak@wikilabels-01:~/datasets$ sudo -u www-data /srv/wikilabels/venv/bin/wikilabels new_campaign kowiki "편집 품질 (예시 2만 개)" da... [16:35:21] 06Revision-Scoring-As-A-Service, 10Wikilabels, 10rsaas-editquality: Deploy editquality campaign in Korean Wikipedia - https://phabricator.wikimedia.org/T161622#3137573 (10Halfak) [16:36:08] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete kowiki editquality campaign - https://phabricator.wikimedia.org/T161627#3137575 (10Halfak) [16:36:13] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete kowiki editquality campaign - https://phabricator.wikimedia.org/T161627#3137589 (10Halfak) [16:36:29] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete editquality campaign for Korean Wikipedia - https://phabricator.wikimedia.org/T161627#3137575 (10Halfak) [16:37:13] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Deploy ORES Review Tool in kowiki - https://phabricator.wikimedia.org/T161628#3137593 (10Halfak) [16:37:40] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train/test damaging and goodfaith models for kowiki - https://phabricator.wikimedia.org/T161629#3137606 (10Halfak) [16:37:46] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete editquality campaign for Korean Wikipedia - https://phabricator.wikimedia.org/T161627#3137575 (10Halfak) [16:37:48] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Train/test damaging and goodfaith models for kowiki - https://phabricator.wikimedia.org/T161629#3137606 (10Halfak) [16:38:26] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels, 10rsaas-editquality: Complete editquality campaign for Korean Wikipedia - https://phabricator.wikimedia.org/T161627#3137621 (10Halfak) [16:38:53] revi, http://labels.wmflabs.org/ui/kowiki/ [16:39:24] I've assigned you to https://phabricator.wikimedia.org/T161627 [16:39:54] Roughly this means, doing some community organizing at kowiki to get people to label edits as "damaging" and/or "goodfaith" so that we can train more advanced models [16:40:31] You can see in the "Task Graph" on phabricator how this related to getting the ORES Review Tool deployed on kowiki :) [16:42:39] ye [16:43:07] I have a herald rule which CCs me when kowiki is in title or desc [16:44:27] and I think translation is done via twn? [17:06:00] halfak: o/ [17:06:29] halfak: I think having looked at "assignments" on https://labels.wmflabs.org/campaigns/wikidatawiki/?campaigns=stats, I assume new campaign participants are blocked from requesting new workset? [17:22:38] glorian_wd, that's right. [17:22:53] Someone is holding onto 8 tasks. [17:23:59] glorian_wd, and they have requested those tasks at some point in the last 2 days. [17:26:11] It's Jklamo [17:57:00] halfak: hi, I haven't forgotten about the code review! I'll try to take a peek today... [17:57:27] awight, great! Thanks. Still a (WIP) for sure, but it works and it's in a good state for strong feedback :) [17:57:39] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Conform ORES sensitivity levels to the new ERI standards - https://phabricator.wikimedia.org/T160575#3137948 (10Catrope) [17:57:48] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Conform ORES sensitivity levels to the new ERI standards - https://phabricator.wikimedia.org/T160575#3104020 (10Catrope) [18:00:33] d'oh! 2kloc of changes [18:00:35] * awight swoons [18:00:46] I like it so far tho [18:00:47] awight, it's not nearly as big as you think. [18:01:04] lost of changes from "context_name, model_names, rev_ids" --> "request" [18:01:13] yeah that's a big win [18:01:16] :) [18:01:26] Check out the scores.py route files too. So much smaller :) [18:01:41] now we can add more nuances w/o an avalanche of spaghetti changes [18:01:49] I had a chat with Amir1 yesterday. I owe you both a textual summary of some stuff. [18:02:09] o interesting [18:02:21] awight, right. I did a couple of those nuances as part of the refactor (though they'll be easy to split later) [18:02:35] E.g. handling precache logging better and implementing v3 of the API. [18:03:13] Minor thing--are Travis tests normally passing? [18:03:36] Hmm... I'm not sure with ORES. That's a good question. [18:04:08] Looks like I broke some of the tests :) [18:04:25] I don't think I want to fix 'em until we're done talking structure though. [18:05:04] I did fully test the functioning of the API though so I know there's nothing crazy/missing. [18:05:14] Hmm... I didn't make sure that celery doesn't get mad though. [18:12:54] https://etherpad.wikimedia.org/p/ores_request_response_refactor [18:26:26] Maybe the world really is this elegant: [18:26:31] > response = self._score(request) [18:26:50] halfak: This change makes me happy! [18:27:57] I'm catching some stuff in my summary! [18:28:59] Next on the gallows: inprogress_results [18:31:35] halfak: is it possible to find who holds the task? [18:31:58] I already found it for you. [18:32:08] oh Jklamo [18:32:21] ok, will follow up him [18:39:53] 06Revision-Scoring-As-A-Service, 10ORES: Implement ScoreRequest/ScoreResponse pattern in ORES - https://phabricator.wikimedia.org/T161526#3138096 (10Halfak) Here are some notes about what's going on here: https://etherpad.wikimedia.org/p/ores_request_response_refactor [18:47:30] halfak: could the workset which currently held by Jklamo be expired after some point of time? So, other editor could claim the workset? [19:19:41] Thanks for taking a pass over it awight :) [19:26:00] Looks like there's no clear conclusion on how one *should* do relative imports. [19:26:21] I'm of the opinion that within-package imports should *always* be relative. [19:26:43] I see that many people are drawing from python 2.5 behavior when recommending against them. [19:27:55] halfak: ? [19:28:00] ? [19:28:17] halfak: could the workset which currently held by Jklamo be expired after some point of time? So, other editor could claim the workset? [19:28:27] 48 hours [19:28:36] Ok thanks [19:30:06] You could start analysis now though. [19:30:24] http://labels.wmflabs.org/campaigns/wikidatawiki/49/?tasks [19:31:25] Wow. One person labeled https://www.wikidata.org/wiki/Q18373187 an "A" and another labeled it an "E" [19:32:39] Same with https://www.wikidata.org/wiki/Q8819500 [19:32:57] The first is much more complete than the 2nf [19:33:42] Looks like there's 0% overlap on A class [19:34:11] 2% overlap on "B" [19:34:27] there's a lot of E's being given [19:34:33] Maybe 5% overlap on "C" [19:35:07] glorian_wd, https://en.wikipedia.org/wiki/Inter-rater_reliability [19:37:03] halfak: oops--Sorry to overstep the "WIP" [19:37:15] awight, did you merge? [19:37:22] yeeah [19:37:30] lemme see if it stuck [19:37:31] :P [19:37:48] whew--seems that it didn't [19:38:00] halfak: hmm. Then we should contact the editors? [19:38:15] halfak: I said "approved", happily that doesn't seem to trigger anything mergey [19:38:28] awight, that's right :) [19:38:40] My thinking is that, tests are failing anyway so there's no great loss, [19:38:42] Approval is welcome :) [19:39:05] and riskier yet, your patch has sprawling implications, so we want to get new work on top of it immediately. [19:39:54] awight, +1 [19:42:06] halfak: BTW, what does it mean by task id: 49? [19:42:59] what meaning are you looking for? [20:06:03] halfak: is it only a campaign ID? or does it signify something? [20:06:08] something else* [20:06:25] Well, there's no task_id: 49 [20:06:31] But there is campaign_id: 49 [20:06:57] oh okay [20:07:27] halfak: should we contact editors who gave very different class to an item? [20:07:49] First, I think you should do your own analysis and write a report. [20:08:25] 06Revision-Scoring-As-A-Service, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), and 2 others: Manage ORES preferences on Watchlist (and Contributions) - https://phabricator.wikimedia.org/T160475#3138306 (10Etonkovidova) I ran the final check on ptwiki... [20:09:30] halfak: analysis about items which have a sharp difference right? [20:10:01] What do you usually write in such labeling result report? [20:41:07] glorian_wd, https://en.wikipedia.org/wiki/Inter-rater_reliability [20:41:44] halfak: gotcha [20:44:42] Do that. Say what we get. Also, everyone loves basic statistics. Like, show me the distribution of the number of labelers and the raw proportions of overlap between them. [20:57:00] halfak: ok got it. I will try [20:57:36] We might have 1 or 2 people who are way off for some reason. [20:58:25] halfak: I had a glance on the data. It seems that most of the editors are confused what scale to give for biological items (e.g. proteins, glucose, etc.) and Wikimedia Disambiguation page [20:59:07] Wikimedia Disambiguation pages are less important that we get right. We should remove those for the next run. [20:59:12] And redirects. [20:59:23] bio items, that should be OK [20:59:32] Those two types of items (i.e. biological items & disambiguation page) generally received very different scales [20:59:36] We just need to get on the same page about what a bio item should look like. [21:00:27] halfak: I think the issue for the bio items are sitelinks. See the discussion on: https://www.wikidata.org/wiki/Wikidata_talk:Item_quality#Pilot_Campaign [21:01:14] Because the bio items usually don't have sitelinks, they got lower grade, even though they have solid references [21:01:22] "I'm not a fan of setting a minimum sitelinks criteria. In this case, there would not be any applicable sitelinks, so zero would be just fine. " [21:01:24] with some translations [21:01:29] Was my contribution to that discussion [21:01:32] yes [21:01:50] I also suggested adding "if applicable" to criterion related to sitelinks [21:02:10] that way, bio items should get fairer quality scale [21:04:08] halfak: take a look at scale A, B, and C. At the moment, those scales enforce sitelinks on items (i.e. items should have sitelinks in order to be in these scales) [21:04:36] (if applicable), right? [21:05:11] halfak: yeah that's my suggestion. But at the moment, we dont have "if applicable" on the criterion related to sitelinks for scale A,B, and C [21:05:39] * glorian_wd check it at : https://www.wikidata.org/wiki/Wikidata:Item_quality [21:05:53] glorian_wd, I agree with you. What do you want me to do? [21:06:22] halfak: nothing for now :P. Perhaps helping me to review the analysis report that I will work on. [21:06:41] probably, I will finish it tomorrow. [21:07:50] halfak: ah! since the page is currently locked for editing, perhaps we should contact Amir1 to help revising the criteria? Or, do we want to unlock the page? [21:08:39] I have jotted down the feedback from labelers regarding to the pilot campaign. Some feedback require us to slightly modify the existing criteria. [22:21:51] 10Revision-Scoring-As-A-Service-Backlog, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Damaging levels on Polish Wikipedia overlap too much - https://phabricator.wikimedia.org/T161655#3138654 (10jmatazzoni) [22:34:13] 10Revision-Scoring-As-A-Service-Backlog, 10Edit-Review-Improvements-RC-Page, 10ORES, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017): Damaging levels on Polish Wikipedia overlap too much - https://phabricator.wikimedia.org/T161655#3138684 (10jmatazzoni) p:05Triage>03High