[00:23:17] 10Scoring-platform-team, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Flagged revs approve model to fiwiki - https://phabricator.wikimedia.org/T166235#3455030 (10awight) >>! In T166235#3449799, @Zache wrote: > > Some multi-diffs examples : [[https://quarry.wmflabs.org/query/20356|Q... [00:36:57] 10Scoring-platform-team-Backlog, 10ORES, 10Documentation, 10I18n: Make ORES documentation translatable - https://phabricator.wikimedia.org/T163786#3455058 (10awight) Something I just thought of--VisualEditor and Page Translation are mutually incompatible: T55974 We should reassess how bad it is once that... [02:14:42] PROBLEM - puppet on ores-worker-08 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:18:18] PROBLEM - puppet on ores-web-03 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:20:42] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:25:44] PROBLEM - puppet on ores-lb-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:32:43] PROBLEM - puppet on ores-redis-02 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:44] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:34:42] PROBLEM - puppet on ores-redis-01 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:09] RECOVERY - puppet on ores-worker-08 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [02:47:40] RECOVERY - puppet on ores-web-03 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [02:50:27] RECOVERY - puppet on ores-worker-07 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:54:27] RECOVERY - puppet on ores-lb-02 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:02:30] RECOVERY - puppet on ores-redis-02 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:04:23] RECOVERY - puppet on ores-redis-01 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:06:40] PROBLEM - puppet on ores-worker-10 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:08:03] PROBLEM - puppet on ores-worker-09 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:12:44] PROBLEM - puppet on ores-worker-06 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:13:46] PROBLEM - puppet on ores-worker-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:14:42] PROBLEM - puppet on ores-worker-08 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:17:20] PROBLEM - puppet on ores-web-03 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:20:59] PROBLEM - puppet on ores-worker-07 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:26:14] PROBLEM - puppet on ores-web-05 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:34:05] RECOVERY - puppet on ores-worker-05 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:36:02] RECOVERY - puppet on ores-worker-10 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:38:22] RECOVERY - puppet on ores-worker-09 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:43:04] RECOVERY - puppet on ores-worker-06 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [03:45:07] RECOVERY - puppet on ores-worker-08 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [03:47:40] RECOVERY - puppet on ores-web-03 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:50:45] RECOVERY - puppet on ores-worker-07 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [03:55:36] RECOVERY - puppet on ores-web-05 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:56:42] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, and 2 others: Add basic bad word check to Wikidata feature set - https://phabricator.wikimedia.org/T170834#3455483 (10Ladsgroup) New models built with adding bad words for English: ``` ScikitLearnClassifier - type: GradientBoos... [08:01:08] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, and 2 others: Add basic bad word check to Wikidata feature set - https://phabricator.wikimedia.org/T170834#3455491 (10Ladsgroup) https://github.com/wiki-ai/editquality/pull/84 [08:28:45] 10Scoring-platform-team-Backlog, 10Wikilabels, 10editquality-modeling, 10Spanish-Sites, 10artificial-intelligence: Complete eswiki edit quality campaign - https://phabricator.wikimedia.org/T131963#3455545 (10MarcoAurelio) Stats update: we're currently at **47%** according to http://labels.wmflabs.org/sta... [10:36:59] 10Scoring-platform-team-Backlog, 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Latvian (lv) - https://phabricator.wikimedia.org/T162031#3455836 (10Papuass) Finally reviewed our words list. Question: if we update it later, is it imported... [10:38:32] 10Scoring-platform-team-Backlog, 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Latvian (lv) - https://phabricator.wikimedia.org/T162031#3455847 (10Papuass) BTW: the link to documentation has been moved but page is not found: https://meta... [10:41:43] 10Scoring-platform-team-Backlog, 10Wikilabels, 10editquality-modeling, 10Spanish-Sites, 10artificial-intelligence: Complete eswiki edit quality campaign - https://phabricator.wikimedia.org/T131963#3455855 (10Elitre) Well done, MarcoAurelio. :) [14:04:23] 10Scoring-platform-team, 10ORES, 10Easy: Swagger docs for v3 interface reference v2 interface - https://phabricator.wikimedia.org/T165839#3456326 (10Halfak) [14:06:50] 10Scoring-platform-team-Backlog, 10ORES, 10Documentation, 10Easy: List ORES use cases - https://phabricator.wikimedia.org/T154440#3456331 (10Halfak) [14:08:16] 10Scoring-platform-team, 10ORES, 10Documentation: Provide a space for reporting bad predictions - https://phabricator.wikimedia.org/T140278#3456333 (10Halfak) [14:08:18] 10Scoring-platform-team, 10ORES, 10Documentation: Provide a space for reporting bad predictions - https://phabricator.wikimedia.org/T140278#2458936 (10Halfak) https://www.mediawiki.org/wiki/ORES/Issues [14:44:25] 10Scoring-platform-team-Backlog, 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for euwiki - https://phabricator.wikimedia.org/T171119#3454661 (10Halfak) p:05Triage>03Normal [14:45:04] 10Scoring-platform-team-Backlog, 10Wikilabels: qunit tests for wikilabels - https://phabricator.wikimedia.org/T171083#3453583 (10Halfak) p:05Triage>03Low [14:45:10] 10Scoring-platform-team-Backlog, 10Wikilabels: qunit tests for wikilabels - https://phabricator.wikimedia.org/T171083#3453583 (10Halfak) p:05Low>03Normal [14:45:18] 10Scoring-platform-team-Backlog, 10Wikilabels: nosetests for wikilabels - https://phabricator.wikimedia.org/T171082#3453570 (10Halfak) p:05Triage>03Normal [14:45:55] 10Scoring-platform-team, 10Wikilabels, 10User-Ladsgroup: linting tests for wikilabels - https://phabricator.wikimedia.org/T171084#3456456 (10Halfak) a:03Ladsgroup [14:46:05] 10Scoring-platform-team, 10Wikilabels, 10User-Ladsgroup: linting tests for wikilabels - https://phabricator.wikimedia.org/T171084#3456457 (10Halfak) p:05Triage>03Low [14:48:01] 10Scoring-platform-team-Backlog, 10ORES, 10monitoring, 10User-Ladsgroup, 10Wikimedia-Incident: Clean up failure ratio monitoring and set up an alarm when it goes more than a certain threshold - https://phabricator.wikimedia.org/T154175#3456463 (10Halfak) [14:49:26] 10Scoring-platform-team-Backlog, 10Technical-Debt: ores-wmflabs-deploy and ores production deploy repos should share a common ancestor - https://phabricator.wikimedia.org/T171014#3450804 (10Halfak) p:05Triage>03Normal [14:50:47] 10Scoring-platform-team-Backlog, 10ORES: ORES deployment finish "successfully" even when uwsgi and celery fail to successfully start up - https://phabricator.wikimedia.org/T170950#3448625 (10Halfak) p:05Triage>03High [14:51:29] 10Scoring-platform-team-Backlog: [Investigate] Hadoop integration for ORES training - https://phabricator.wikimedia.org/T170650#3456477 (10Halfak) p:05Triage>03Low [14:51:56] 10Scoring-platform-team-Backlog, 10Analytics, 10revscoring, 10artificial-intelligence: [Investigate] Hadoop integration for ORES training - https://phabricator.wikimedia.org/T170650#3438153 (10Halfak) [14:54:26] 10Scoring-platform-team-Backlog, 10Wikimania-Hackathon-2017, 10Documentation: Improve documentation about how to install ORES - https://phabricator.wikimedia.org/T170506#3456509 (10Halfak) p:05Triage>03Normal [14:54:41] 10Scoring-platform-team-Backlog, 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Complete English Wikipedia edit quality campaign (v2) - https://phabricator.wikimedia.org/T163464#3456512 (10Halfak) p:05Triage>03Low [14:56:31] 10Scoring-platform-team-Backlog, 10editquality-modeling, 10artificial-intelligence: Train/test TensorFlow goodfaith/damaging models for enwiki - https://phabricator.wikimedia.org/T161380#3456524 (10Halfak) p:05Normal>03Lowest [14:56:36] 10Scoring-platform-team-Backlog, 10editquality-modeling, 10artificial-intelligence: Train/test TensorFlow goodfaith/damaging models for fawiki - https://phabricator.wikimedia.org/T161381#3456529 (10Halfak) p:05Normal>03Lowest [14:56:49] 10Scoring-platform-team-Backlog, 10Wikidata, 10editquality-modeling, 10artificial-intelligence: Train/test TensorFlow goodfaith/damaging models for wikidatawiki - https://phabricator.wikimedia.org/T161382#3456531 (10Halfak) p:05Normal>03Lowest [14:56:54] 10Scoring-platform-team-Backlog, 10editquality-modeling, 10artificial-intelligence: Train damaging/goodfaith model for English Wiktionary - https://phabricator.wikimedia.org/T170487#3433018 (10Halfak) p:05Triage>03Low [14:57:46] 10Scoring-platform-team-Backlog, 10revscoring, 10artificial-intelligence: Build TensorFlow models support in revscoring - https://phabricator.wikimedia.org/T161377#3456540 (10Halfak) [15:00:08] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Improve cleaning of article quality assessment datasets - https://phabricator.wikimedia.org/T170434#3456556 (10Halfak) [15:03:09] 10Scoring-platform-team-Backlog, 10ORES: ORES UI could accept more human-readable inputs such as pasted article links - https://phabricator.wikimedia.org/T170168#3421485 (10Halfak) Maybe a UI that does this: https://commons.wikimedia.org/wiki/File:Enwiki.biology.monthly_wp10.svg [15:05:10] 10Scoring-platform-team-Backlog, 10artificial-intelligence: Draft specification of article importance API - https://phabricator.wikimedia.org/T167362#3330642 (10Halfak) p:05Triage>03Low [15:06:53] 10Scoring-platform-team-Backlog, 10Wikimania-Hackathon-2017: ORES @ the Wikimania Hackathon - https://phabricator.wikimedia.org/T170015#3416777 (10Halfak) p:05Triage>03High [15:07:47] 10Scoring-platform-team-Backlog, 10ORES, 10Operations, 10Graphite, 10User-fgiunchedi: Regularly purge old ores graphite metrics - https://phabricator.wikimedia.org/T169969#3415026 (10Halfak) p:05Normal>03High [15:08:26] 10Scoring-platform-team-Backlog, 10ORES, 10Operations, 10Graphite, 10User-fgiunchedi: Regularly purge old ores graphite metrics - https://phabricator.wikimedia.org/T169969#3415026 (10Halfak) I think we'd like to keep some high level metrics forever, others for just 90 days and many for just 30 days. Is... [15:09:28] 10Scoring-platform-team-Backlog, 10ORES, 10Wikimedia-Logstash: Send celery logs and events to logstash - https://phabricator.wikimedia.org/T169586#3456599 (10Halfak) p:05Triage>03High [15:13:52] 10Scoring-platform-team-Backlog, 10revscoring, 10artificial-intelligence: Use multithreading in test_model - https://phabricator.wikimedia.org/T169843#3410274 (10Halfak) [15:14:09] 10Scoring-platform-team-Backlog, 10revscoring, 10artificial-intelligence: Use multithreading in test_model - https://phabricator.wikimedia.org/T169843#3410274 (10Halfak) p:05Triage>03Low [15:14:12] 10Scoring-platform-team-Backlog, 10ORES, 10Operations, 10Graphite, 10User-fgiunchedi: Regularly purge old ores graphite metrics - https://phabricator.wikimedia.org/T169969#3415026 (10ori) Take a look at https://github.com/wikimedia/puppet/blob/c2543d7f80fefbe39901897882c60d91d98c3950/modules/role/manifes... [15:14:33] 10Scoring-platform-team-Backlog, 10ORES, 10Patch-For-Review: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3456616 (10Halfak) p:05Triage>03High [15:15:18] 10Scoring-platform-team, 10ORES, 10Patch-For-Review: Stress/capacity test new ores* cluster - https://phabricator.wikimedia.org/T169246#3391572 (10Halfak) [15:17:04] 10Scoring-platform-team-Backlog, 10draftquality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly draft quality dataset on labsDB - https://phabricator.wikimedia.org/T167697#3341451 (10Halfak) p:05Triage>03Low [15:17:55] 10Scoring-platform-team-Backlog, 10draftquality-modeling, 10artificial-intelligence: Generate a draft quality dataset - https://phabricator.wikimedia.org/T171178#3456638 (10Halfak) [15:18:10] 10Scoring-platform-team-Backlog, 10draftquality-modeling, 10artificial-intelligence: [Discuss] Hosting the draft quality dataset on labsDB - https://phabricator.wikimedia.org/T167697#3341451 (10Halfak) [15:18:56] 10Scoring-platform-team-Backlog, 10Wikidata, 10artificial-intelligence: [Spike] Use suggested properties to get signal for completeness - https://phabricator.wikimedia.org/T158430#3036741 (10Halfak) p:05Normal>03Low [15:19:45] 10Scoring-platform-team, 10draftquality-modeling, 10artificial-intelligence: [Discuss] draftquality on a sample, humongous everything, or something else? - https://phabricator.wikimedia.org/T168909#3456664 (10Halfak) [15:20:39] 10Scoring-platform-team, 10draftquality-modeling, 10artificial-intelligence: [Discuss] draftquality on a sample, humongous everything, or something else? - https://phabricator.wikimedia.org/T168909#3380334 (10Halfak) We that we have {T169809}, we can run more parallel jobs on a large training set. [15:22:19] 10Scoring-platform-team-Backlog: Design how we'll train models which depend on private data - https://phabricator.wikimedia.org/T168908#3380311 (10Halfak) The new stat machines are ready. See {T165366}. We should be able to start training models there. Note that this machine is in the prod cluster -- all who... [15:24:04] 10Scoring-platform-team-Backlog: Design how we'll train models which depend on private data - https://phabricator.wikimedia.org/T168908#3456702 (10Halfak) We'll need to double check our enchant dictionaries. They are the only thing that is *really* OS dependent. [15:26:13] 10Scoring-platform-team-Backlog, 10editquality-modeling, 10artificial-intelligence: [Spec] Bury horrors of the editquality makefile - https://phabricator.wikimedia.org/T168455#3456729 (10Halfak) [15:26:57] 10Scoring-platform-team-Backlog, 10editquality-modeling, 10artificial-intelligence: [Spec] Bury horrors of the editquality makefile - https://phabricator.wikimedia.org/T168455#3364890 (10Halfak) p:05Triage>03Low [15:27:23] 10Scoring-platform-team-Backlog, 10ORES, 10revscoring, 10artificial-intelligence: Consider adding a model to ORES for predicting which pages should be protected - https://phabricator.wikimedia.org/T149677#3456737 (10Halfak) p:05Normal>03Low [15:27:43] 10Scoring-platform-team-Backlog, 10ORES, 10RESTBase, 10RESTBase-API, 10Services (next): Use RESTBase for ORES precaching - https://phabricator.wikimedia.org/T166161#3456738 (10Halfak) p:05Normal>03Low [17:00:36] o/ [17:19:13] 10Scoring-platform-team-Backlog, 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Latvian (lv) - https://phabricator.wikimedia.org/T162031#3457120 (10Halfak) Great! Thank you. We don't update automatically based on edits to those pages.... [17:19:21] 10Scoring-platform-team-Backlog, 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Latvian (lv) - https://phabricator.wikimedia.org/T162031#3457121 (10Halfak) [17:19:31] 10Scoring-platform-team, 10Bad-Words-Detection-System, 10revscoring, 10User-Ladsgroup, 10artificial-intelligence: Add language support for Latvian (lv) - https://phabricator.wikimedia.org/T162031#3150287 (10Halfak) [17:51:52] nice work, Amir1 :) [17:52:17] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, and 2 others: Add basic bad word check to Wikidata feature set - https://phabricator.wikimedia.org/T170834#3457244 (10Halfak) > I saw was 5% increase in recall at 90% precision. That's pretty good. [17:53:46] halfak: Thanks <3 [17:53:58] halfak: what do you think about next steps for https://github.com/wiki-ai/wikilabels/pull/195 ? [17:54:56] Amir1, good Q. I'm still confused about how that worked in the past. [17:55:05] Unrelated, I just came across this: https://commons.wikimedia.org/wiki/File:Aaron_Halfaker,_Burritoface.jpg [17:55:44] that's a good management pic :D [17:56:09] You made my day :)))) [17:56:20] well, what's left of it :D [17:56:41] Amir1, do you know how that var got populated in the past? [17:56:46] $html_entity [17:57:05] I tried to do search in old version but no success [17:57:22] nothing came up [17:57:53] overall $ confused me too (I think there are more variables that are not jquery class and have $ in the first) [17:58:41] I'm talking about https://stackoverflow.com/questions/205853/why-would-a-javascript-variable-start-with-a-dollar-sign [18:05:16] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Use 'informals', 'badwords', etc. in Wikidata feature set - https://phabricator.wikimedia.org/T162617#3457311 (10Ladsgroup) Should we call this done? or set it back for more signals in later tries? [18:09:14] i HATE anything java its syntax is a pain [18:17:50] halfak: renamed the variable [18:28:10] 10Scoring-platform-team-Backlog, 10ORES, 10Scap: ORES deployment finish "successfully" even when uwsgi and celery fail to successfully start up - https://phabricator.wikimedia.org/T170950#3457373 (10Ladsgroup) [19:09:34] Amir1, I forget. Did you confirm that there's a strong norm in javascript or amongst wikimedia devs that you shouldn't accept all parameters when passing a function as an argument? [19:09:49] I just missed awight|lunch [19:10:42] halfak: Yes, it's the norm, at least everywhere I saw javascript (mediawiki, wikibase, etc.) [19:11:08] gotcha. OK then. [19:11:37] I was ready to look at finish stuff. So I think I might go AFK for a bit. I'm surprisingly tired and I'm going to try a nap. I'll be AFK for about an hour [19:11:42] *Finnish [19:12:24] halfak: Thanks! [19:12:30] Next step, stylelint [19:15:12] 10Scoring-platform-team, 10ORES, 10Documentation: Provide a space for reporting bad predictions - https://phabricator.wikimedia.org/T140278#3457536 (10Halfak) 05Open>03Resolved [19:15:18] 10Scoring-platform-team, 10ORES, 10Easy: Swagger docs for v3 interface reference v2 interface - https://phabricator.wikimedia.org/T165839#3457538 (10Halfak) 05Open>03Resolved [19:15:26] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, and 2 others: Add basic bad word check to Wikidata feature set - https://phabricator.wikimedia.org/T170834#3457539 (10Halfak) 05Open>03Resolved [19:15:28] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Use 'informals', 'badwords', etc. in Wikidata feature set - https://phabricator.wikimedia.org/T162617#3457540 (10Halfak) [19:15:33] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, and 2 others: Add entropy-related and uppercase-related measures to comments - https://phabricator.wikimedia.org/T170835#3457541 (10Halfak) 05Open>03Resolved [19:15:35] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Use 'informals', 'badwords', etc. in Wikidata feature set - https://phabricator.wikimedia.org/T162617#3168776 (10Halfak) [19:56:31] halfak: I'm around if you want to go over the fiwiki.flaggedrevs plan [19:56:57] laptop drive is DOA btw, so I'll be distracted rebuilding a machine this afternoon [19:57:02] sux. [20:28:00] o/ [20:28:03] awight, back [20:28:25] Oh no! the new drive is dead too? [20:29:45] naw, but what I tried to do was swap the SSD chips from the old machine to a loaner, in case it was a hw failure of something other than the disks [20:29:50] it turned out to be my disks. [20:30:18] oh! Gotcha. [20:30:26] DOA = Dead on arrival. [20:30:52] Regardless, want to talk fiwiki or work on that. Either is OK with me. [20:31:26] talking would be best. hangouts? [20:37:05] halfak who you buy from [20:59:46] hmm.... This is a redirect that has a predicted ORES score of FA: https://en.wikipedia.org/w/index.php?title=Keita_dynasty&oldid=778001390 [21:00:01] https://ores.wikimedia.org/v2/scores/enwiki/wp10/778001390?features [21:00:18] ragesoss, known issue. Ask Nettrom :) [21:01:04] halfak: cool beans. my intern Keerthana will probably show up and say hi and some point. [21:01:20] \o/ great! [21:01:25] she's working on the project to provide suggestions in the dashboard, some based on ORES data. [21:02:02] she came across that yesterday. [21:04:27] Nettrom: is there an issue for that somehwere we can subscribe to? (redirects predicted as 40% FA) [21:04:37] yes, hang on a sec [21:05:03] T170434 [21:05:03] T170434: Improve cleaning of article quality assessment datasets - https://phabricator.wikimedia.org/T170434 [21:11:44] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Improve cleaning of article quality assessment datasets - https://phabricator.wikimedia.org/T170434#3457969 (10Ragesoss) @Keer25: Nettrom is the main dev for the wp10 model, and he pointed me here when I noted that issue of the re... [21:12:31] Nettrom: is there a pipeline for human feedback on wp10 predictions, to improve the model? [21:13:08] and if not, what scale of data would be necessary to make that kind of thing useful? [21:16:21] ragesoss: No pipeline at the moment as far as I know. With regards to scale, let me go back and look at the first paper we wrote about this, might get some clues there. [21:18:58] Hmmm, I could also do some tests with various training set sizes once I’ve cleaned up the data gathering code… we might be able to get decent performance out of a dataset with say 500 articles from each class (3,000 in total) [21:21:06] I'm asking because I have a UI now for putting ORES-based data in front of users, and it wouldn't be that hard to collect some kind of signal from users about that prediction, like 'that seems to high / too low/ about right' [21:21:54] my users are not familiar enough with Wikipedia conventions to provide accurate from-scratch ratings (vs. the natural data you based the current model on) [21:23:11] but if some kind of basic feedback about the model scores could help refine it, I'd be interested in talking with you about it. [21:23:16] I wonder if what would be most useful is flagging predictions that appear to be way off [21:23:51] one of the challenges with the predictions that we’re doing is to be able to identify when they’re off by more than a class [21:27:48] Hmmm, I’ll need to think about this [21:38:22] we had ~8500 assigned articles this most recent term, and it'll probably be a little higher next term, to give you an idea of the scale of data it might be possible to collect. [21:39:07] I think I could put something like a 'this rating seems way too high' or 'this rating seems way to low' widget in and collect that kind of outlier data without much work. [21:40:02] so if you get to the point where you're itching for some kind of human feedback on the ORES predictions as a regular trickle, give me a ping. [21:40:38] wiki-ai/revscoring#1112 (import_typo - 69ab465 : Adam Roses Wight): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/255836560 [21:40:56] ragesoss, sorry was AFK, but what who would be generating the feedback? [21:41:13] Also, we're building a system for better capturing feedback, so we'll get back to you on that :) [21:42:39] halfak: students who are assigned to improve the article, and/or instructors who are selecting articles in the course topic area for students to improve. [21:43:16] Hmm... not sure we'd use their feedback in training the model. [21:44:41] halfak: yeah, that's reasonable. I was thinking mainly in terms of identifying cases were the model clashes significantly with what (somewhat noob-ish) users perceive. [21:45:01] Right. That's something I want to measure and track. :) [21:45:12] The new feedback system will be very permissive [21:45:29] cool beans. [21:45:31] :) [21:46:35] Essentially, we'll give you an API endpoint. So long as someone is logged in (OAuth) and we can ID them, you'll be able to have them submit feedback to the endpoint. [21:46:48] And we'll aggregate it into a big queryable database. [21:47:11] "Show me all the places where newcomers disagreed with ORES" [21:48:42] oh, cool. [21:49:42] are there UI plans for putting that in front of large numbers, or would this be just through a dedicated feedback interface on-wiki (plus whatever comes through the API)? [21:51:02] No UI on our side. [21:51:20] We'd expose the API to devs and get the data in places that could be queried (e.g. quarry) [21:52:25] We'd provide a basic schema for what we expect as feedback and leave it to you to set up a simple UI for letting a user provide it. We might also give you a simple HTML blob or some OOJS UI code. [21:52:43] So you wouldn't have to track changes to our schema. [21:54:26] sounds great. do you have a sketch of what that schema will look like, or is that still TBD? [21:57:01] Schema and implementation proposals are due by the end of this quarter. [21:57:39] Likely first submissions of misclassifications will happen in the winter. [21:58:16] will it also be able to store free-form feedback about what's wrong with an article? [23:03:00] awight hi