[00:03:21] 06Revision-Scoring-As-A-Service, 10revscoring: Implement abstraction for Sparse Feature Vectors - https://phabricator.wikimedia.org/T132580#2581034 (10Halfak) More cleanup and I've implemented the vector pattern in the tests for scoring models. I think that all that really remains for this work is setting up... [00:07:46] 06Revision-Scoring-As-A-Service, 10revscoring, 07Spike: [Spike] Investigate HashingVectorizer - https://phabricator.wikimedia.org/T128087#2581049 (10Halfak) I'm getting pretty close with T132580. I'll likely want to use this work to generate a demo that the system works. I'll likely pick that up the next t... [00:11:09] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 07Easy, 15User-Ladsgroup: Unclear header for ORES options in Preferences - https://phabricator.wikimedia.org/T143822#2581058 (10kaldari) I guess this really only matters once ORES is no longer a beta feature (since you have to activate ORES to s... [00:15:30] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 07Easy, 15User-Ladsgroup: Unclear header for ORES options in Preferences - https://phabricator.wikimedia.org/T143822#2580206 (10Legoktm) I honestly doubt people will understand what "Objective Revision Evaluation Service" is either. [00:18:52] 06Revision-Scoring-As-A-Service, 10Wikimedia-Site-requests, 07Community-consensus-needed: Enable RecentChanges patrolling on English Wikipedia - https://phabricator.wikimedia.org/T143791#2581067 (10Legoktm) [00:50:16] 06Revision-Scoring-As-A-Service, 10revscoring, 07Spike: [Spike] Investigate HashingVectorizer - https://phabricator.wikimedia.org/T128087#2581106 (10Sabya) I am also pretty close to evaluating the first model with combined features😃. [01:01:28] 06Revision-Scoring-As-A-Service, 10Wikimedia-Site-requests, 07Community-consensus-needed: Enable RecentChanges patrolling on English Wikipedia - https://phabricator.wikimedia.org/T143791#2579046 (10Dereckson) >>! In T143791#2579072, @Ladsgroup wrote: > One quick note: the patoller is not an extension. It's i... [01:09:50] 06Revision-Scoring-As-A-Service, 10Wikimedia-Site-requests, 07Community-consensus-needed: Enable RecentChanges patrolling on English Wikipedia - https://phabricator.wikimedia.org/T143791#2581135 (10Dereckson) @Halfak You probably need to start a https://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment ac... [01:17:22] 06Revision-Scoring-As-A-Service, 10Wikimedia-Site-requests, 07Community-consensus-needed: Enable RecentChanges patrolling on English Wikipedia - https://phabricator.wikimedia.org/T143791#2581143 (10MusikAnimal) Is there a permission that will prevent a user's edits from being flagged by ORES? I would guess `... [01:21:01] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 07Easy, 15User-Ladsgroup: Unclear header for ORES options in Preferences - https://phabricator.wikimedia.org/T143822#2581150 (10kaldari) Revision scoring? [14:01:57] halfak: I will join in a sec [14:02:00] kk [14:12:48] 06Revision-Scoring-As-A-Service, 10MediaWiki-API, 10MediaWiki-extensions-ORES, 07Epic: [Epic] Implement ORES service proxy in api.php - https://phabricator.wikimedia.org/T143895#2582434 (10Halfak) [14:26:06] 10Revision-Scoring-As-A-Service-Backlog, 06Collaboration-Team-Triage, 10Edit-Review-Improvements, 10MediaWiki-extensions-ORES: Include goodfaith model information in ORES review tool - https://phabricator.wikimedia.org/T137966#2582502 (10Halfak) [14:29:20] 10Revision-Scoring-As-A-Service-Backlog, 06Research-and-Data: Edit type classification lit review - https://phabricator.wikimedia.org/T107383#2582526 (10Halfak) p:05Normal>03Lowest [14:29:42] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Expose ores_model data in API using meta=ores - https://phabricator.wikimedia.org/T143617#2582529 (10Halfak) p:05Triage>03Normal [14:30:13] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Introduce rcshow=oresreview and similar ones - https://phabricator.wikimedia.org/T143616#2573460 (10Halfak) p:05Triage>03Normal [14:30:20] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Introduce ORES rvprop - https://phabricator.wikimedia.org/T143614#2573432 (10Halfak) p:05Triage>03Normal [14:33:05] 06Revision-Scoring-As-A-Service, 10MediaWiki-API, 10MediaWiki-extensions-ORES, 07Epic: [Epic] Implement ORES service proxy in api.php - https://phabricator.wikimedia.org/T143895#2582539 (10Halfak) [14:33:56] 06Revision-Scoring-As-A-Service, 10MediaWiki-API, 10MediaWiki-extensions-ORES, 07Epic: [Epic] Implement ORES service proxy in api.php - https://phabricator.wikimedia.org/T143895#2582434 (10Halfak) [14:33:59] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Build an entry point to store scores in ORES extension - https://phabricator.wikimedia.org/T131785#2582545 (10Halfak) [14:43:12] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Embed machine readable ores scores as data on pages where ORES scores things - https://phabricator.wikimedia.org/T143611#2582561 (10Halfak) [14:43:30] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Embed machine readable ores scores as data on pages where ORES scores things - https://phabricator.wikimedia.org/T143611#2573377 (10Halfak) @Legoktm, I just made some updates. Does this make the task more clear? [14:45:19] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Embed machine readable ores scores as data on pages where ORES scores things - https://phabricator.wikimedia.org/T143611#2582569 (10Halfak) [14:47:59] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Request scores when someone checks out edits that are not stored in ores_classification - https://phabricator.wikimedia.org/T143612#2582584 (10Halfak) p:05Triage>03Normal [14:48:06] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Embed machine readable ores scores as data on pages where ORES scores things - https://phabricator.wikimedia.org/T143611#2573377 (10Halfak) p:05Triage>03Normal [14:48:20] 10Revision-Scoring-As-A-Service-Backlog, 10Wikilabels: Metrics collection for Wikilabels - https://phabricator.wikimedia.org/T132295#2582589 (10Halfak) p:05Normal>03Low [14:48:43] 10Revision-Scoring-As-A-Service-Backlog, 10ORES: [Spec] Celery worker monitoring - https://phabricator.wikimedia.org/T139948#2582590 (10Halfak) p:05Normal>03Low [14:52:53] 06Revision-Scoring-As-A-Service, 06Research-and-Data-Backlog, 10rsaas-editquality, 07Epic, 03Research-and-Data-2016-Q3: [Epic] Explore disparate impacts of damage detection and goodfaith prediction on anons and newcomers. - https://phabricator.wikimedia.org/T120138#2582624 (10Halfak) Marking this as reso... [14:53:00] 06Revision-Scoring-As-A-Service, 06Research-and-Data-Backlog, 10rsaas-editquality, 07Epic, 03Research-and-Data-2016-Q3: [Epic] Explore disparate impacts of damage detection and goodfaith prediction on anons and newcomers. - https://phabricator.wikimedia.org/T120138#2582626 (10Halfak) 05Open>03Resolved [14:55:16] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-articlequality , 03Research-and-Data-2017-Q1: Generate recent article quality scores for English Wikipedia - https://phabricator.wikimedia.org/T135684#2582630 (10Halfak) [14:55:45] 06Revision-Scoring-As-A-Service, 10rsaas-articlequality , 03Research-and-Data-2017-Q1: Generate recent article quality scores for English Wikipedia - https://phabricator.wikimedia.org/T135684#2306938 (10Halfak) [14:56:52] 06Revision-Scoring-As-A-Service, 10rsaas-articlequality , 03Research-and-Data-2017-Q1: Generate recent article quality scores for English Wikipedia - https://phabricator.wikimedia.org/T135684#2306938 (10Halfak) a:03Halfak [15:18:01] ^ that looks like a backlog grooming :D [15:18:28] halfak: do you have a suggestion for this? https://phabricator.wikimedia.org/T143822 [15:18:51] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 07Easy, 15User-Ladsgroup: Unclear header for ORES options in Preferences - https://phabricator.wikimedia.org/T143822#2582687 (10Ladsgroup) @kaldari What about "ORES review tool"? [15:19:56] Yeah. I don't think that Objective Revision Evaluation Service is more descriptive really. [15:24:26] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES, 07Easy, 15User-Ladsgroup: Unclear header for ORES options in Preferences - https://phabricator.wikimedia.org/T143822#2582712 (10Halfak) I like the term "Revision scoring" as it implies what's being done. I don't think that expanding ORES' acron... [15:50:45] (03PS1) 10Ladsgroup: Change ORES header to "Revision scoring" in preferences [extensions/ORES] - 10https://gerrit.wikimedia.org/r/306685 (https://phabricator.wikimedia.org/T143822) [16:08:40] halfak: one question. Are you okay with the deployment window one hour before our weekly sync? [16:08:59] Yes [16:09:03] I'm booking a deployment window [16:09:06] kk [16:09:20] Can we do both the service and the extension at the same time? [16:11:55] yes, that's my plan [16:12:06] (I will do the deployments) [16:16:44] (03CR) 10Thiemo Mättig (WMDE): [C: 031] Change ORES header to "Revision scoring" in preferences [extensions/ORES] - 10https://gerrit.wikimedia.org/r/306685 (https://phabricator.wikimedia.org/T143822) (owner: 10Ladsgroup) [16:26:40] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: ORES extension should Assume good faith page creator's revisions - https://phabricator.wikimedia.org/T137846#2582838 (10Ladsgroup) > Hi, in Russia Wikipedia we have 'editor' and 'autoeditor' users . These users almost the same with 'reviewer' and 'a... [16:38:19] halfak: https://grafana.wikimedia.org/dashboard/db/ores [16:38:35] It seems we can handle up to 60K / min now :D [16:43:08] (03PS1) 10Ladsgroup: Fix CheckModelVersions by changing order of actions [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/306689 (https://phabricator.wikimedia.org/T143799) [16:45:03] (03PS1) 10Ladsgroup: Fix for purging scores [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/306691 (https://phabricator.wikimedia.org/T143798) [16:45:42] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Add 'editor' and 'autoeditor' user groups to ruwiki models - https://phabricator.wikimedia.org/T143909#2582897 (10Halfak) [16:46:02] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: ORES extension should Assume good faith page creator's revisions - https://phabricator.wikimedia.org/T137846#2582911 (10Halfak) +1 I made a task: T143909 [16:54:36] 06Revision-Scoring-As-A-Service, 10Wikimedia-Site-requests, 07Community-consensus-needed: Enable RecentChanges patrolling on English Wikipedia - https://phabricator.wikimedia.org/T143791#2582952 (10Halfak) @Dereckson, I'm aware of that, thanks. The feature was requested by someone else. I just filed the bu... [17:00:06] Amir1, woah! [17:00:08] Cool! [18:07:23] (03CR) 10Thcipriani: [C: 032] "SWAT" [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/306689 (https://phabricator.wikimedia.org/T143799) (owner: 10Ladsgroup) [18:08:24] (03CR) 10Thcipriani: [C: 032] "SWAT" [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/306691 (https://phabricator.wikimedia.org/T143798) (owner: 10Ladsgroup) [18:13:50] (03Merged) 10jenkins-bot: Fix CheckModelVersions by changing order of actions [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/306689 (https://phabricator.wikimedia.org/T143799) (owner: 10Ladsgroup) [18:17:16] (03Merged) 10jenkins-bot: Fix for purging scores [extensions/ORES] (wmf/1.28.0-wmf.16) - 10https://gerrit.wikimedia.org/r/306691 (https://phabricator.wikimedia.org/T143798) (owner: 10Ladsgroup) [18:48:40] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Embed machine readable ores scores as data on pages where ORES scores things - https://phabricator.wikimedia.org/T143611#2583504 (10Legoktm) Yeah, thanks. I don't think wbEntity is a good idea - there's a bug somewhere about how it needs to... [19:03:31] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Embed machine readable ores scores as data on pages where ORES scores things - https://phabricator.wikimedia.org/T143611#2583560 (10Halfak) @Legoktm, if not JSON, how would you propose we serialize the data? Is there another format that you... [19:05:33] 10Revision-Scoring-As-A-Service-Backlog, 10MediaWiki-extensions-ORES: Embed machine readable ores scores as data on pages where ORES scores things - https://phabricator.wikimedia.org/T143611#2583564 (10Legoktm) Just multiple data attributes? data-ores-damaging, data-ores-reverted, etc.? [19:06:54] halfak: I think embedding JSON in html attributes gets weird because the " have to be escaped and it just looks ugly [19:07:23] Hey legoktm. Agreed, but what's the alternative? [19:07:43] multiple data attributes? [19:07:51] data-ores-damaging, data-ores-reverted, etc.? [19:07:57] unless there's more info you want to expose? [19:11:10] Yeah... each one of those attributes has a complex value [19:11:17] I guess it doesn't have to [19:11:18] hmmm [19:12:59] I thought it's just a number? [19:14:08] Well... so, each model has a set of numbers. [19:14:20] Soon we'll have a multi-class problem: Article quality prediction [19:14:52] there, a prediction looks like: {FA: 5%, GA: 10%, B: 70%, C: 15%, ...} [19:17:16] ...JSON sounds good then [19:18:30] yeah... Maybe if there were "JSON, but with single quotes" ;) [19:18:46] To avoid the encode-splosion [19:26:00] 06Revision-Scoring-As-A-Service, 10MediaWiki-extensions-ORES: ORES extension should Assume good faith page creator's revisions - https://phabricator.wikimedia.org/T137846#2583582 (10Iniquity) @Ladsgroup, RuCommunity disabled a 'patrol' flag a lot of time ago:) @Halfak, @Ladsgroup: thx, I think this is the bes... [20:19:48] how can I debug why ORES detected a revision as bad? [20:28:21] Platonides, that's a hard question to answer, but there are some things that I can show you. [20:28:28] Can you share the revid with me? [20:29:09] Amir1, hey dude. Just wanted to let you know that I'm picking up the "editor", "autoeditor" task for ruwiki [20:29:33] halfak: awesome :) [20:30:15] Once I'm done with a wikidata patch I do the beta wiki part :) [20:30:32] sure: https://es.wikipedia.org/w/index.php?diff=93176830&oldid=92151054 [20:30:49] PatruBOT reverted because ORES said so :P [20:31:28] the user is wrong in that tanques is indeed appropiate [20:31:38] but the edit is good [20:31:54] and I see nothing likely to trigger a vandalism detection [20:32:36] Platonides, I'm looking at https://ores.wikimedia.org/v2/scores/eswiki/reverted/93176830?features [20:32:37] Platonides: hola! [20:32:38] https://ores.wikimedia.org/v2/scores/eswiki/?models=reverted&revids=93176830 [20:32:42] Which outputs the predictive features [20:32:48] hey Amir1 [20:33:03] :) [20:33:05] DarTar: Hey, how are you? [20:33:06] did you want me to retweet the enwiki beta rollout announcement? [20:33:14] what's false and true there, Amir1 ? [20:33:15] link? [20:33:31] DarTar: yeah, I want you to tweet about it. I can retweet [20:33:37] or vice versa, your call :) [20:33:51] do it and I’ll give you some @WikiResearch love [20:34:05] :D [20:34:06] sure [20:34:16] add a screenshot from the beta settings maybe [20:34:45] Platonides, true == probability that the edit will need to be reverted [20:34:55] false = the probability that it will not. [20:35:07] Platonides, I'm not seeing an obvious reason that the edit scored so high. [20:35:15] So now I'm going to try changing some data for the model. [20:35:46] If the editor were registered, the score would be a little bit lower. https://ores.wikimedia.org/v2/scores/eswiki/reverted/93176830?features&feature.revision.user.is_anon=false [20:35:55] true = 0.794 [20:36:40] yeah, that's what surprised me [20:36:45] that it was quite innocent [20:38:07] So, making it so that the editor removed no dictionary words actually increases the "true" prediction [20:38:08] https://ores.wikimedia.org/v2/scores/eswiki/reverted/93176830?features&feature.revision.user.is_anon=false&feature.spanish.dictionary.revision.diff.dict_word_delta_decrease=0&feature.spanish.dictionary.revision.diff.dict_word_prop_delta_decrease=0 [20:38:14] true = 0.82 [20:40:54] DarTar: https://twitter.com/AmirSarabadani/status/768910733757210624 [20:41:28] Amir1: brilliant [20:41:30] Platonides, I'm drawing a blank. I'm not sure what made this edit get flagged. [20:42:08] Amir1 let me add a screenshot too [20:42:20] Awesome [20:42:23] thanks :) [20:43:44] Platonides, it would be great if we had the damaging/goodfaith models live [20:44:08] I bet there's some peculiar pattern in reverts used to train the "reverted" model that is getting picked up here. [20:44:14] Do you see any trends in what gets flagged? [20:54:13] it's the first one I look at [20:54:31] I happened to see the edit [20:54:38] the summary picked my attention [20:54:49] and was leaving a message to the user when the bot beated me to it [20:54:55] saying that it was bad [21:24:38] Platonides, does the bot automatically revert? [21:25:09] That might be a little bit troublesome with the "reverted" model. It's not as good as what we can do once the wiki labels campaign is done. [21:25:43] https://es.wikipedia.org/wiki/Wikipedia:Etiquetando/Valorar_calidad [21:26:15] It looks like we have 1210/8434 labels [21:26:33] halfak: yes [21:27:37] you should probably treat differently all PatruBOT edits from June on [21:28:34] otherwise you will only be reinforcing your own -maybe mistaken- training [21:28:34] OTOH if the revert was right noone would need to change anything [21:28:34] a revert of the revert is a stronger signal [21:28:46] supposing it's not just the original editor [21:29:06] additionally, older reverts by PatruBOT were also automated [21:29:12] although using a different algorithm [21:29:33] which may be easier to copy based on the code rather than training [21:30:08] Platonides, we'll be looking for new data from wiki labels, but given the effect that ORES is likely to have, we'll never gather a new sample for the "reverted" models. [21:30:16] Then again, we do handle the re-reverted case [21:30:24] We also exclude self-reverts [21:40:35] OK. Extracting features for ruwiki. [21:41:30] Amir1, I ended up reworking our user-group sets entirely, so I'll be rebuilding the full set of models. I'll start with the ruwiki models first though. [21:41:36] I'll have a WIP PR for you. [21:42:00] Oh, All wikis? [21:42:27] Yeah. It'll affect all of them. I'll show you in a moment. [21:44:33] Amir1, https://github.com/wiki-ai/editquality/pull/45/files [21:44:33] I reviewed user groups and what rights they apply for a bunch of wikis [21:44:33] yes, it's awesome [21:44:46] Cool :) [21:44:59] Hopefully, we'll see a fitness boost for ruwiki [21:45:05] If so, I'll apply this all across the board. [21:45:19] And I expect it will be easier than last time since I cleaned up the makefile ^_^ [21:50:51] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Extend user group features - https://phabricator.wikimedia.org/T143909#2584160 (10Halfak) [21:54:32] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Extend user group features - https://phabricator.wikimedia.org/T143909#2584175 (10Halfak) https://github.com/wiki-ai/editquality/pull/45 Here's the new classes and what they match: * __revision.user.is_bot__: 'bot' * __revision.user.has_advanced_... [22:06:55] halfak: tell me when it's not WIP anymore so I review :) [22:07:52] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Extend user group features - https://phabricator.wikimedia.org/T143909#2584231 (10Iniquity) I think a 'ipblock-exempt' flag is not trusted. This flag can be given to people who have never worked with articles. [22:11:37] Amir1, ETA tomorrow :) [22:13:45] 10Revision-Scoring-As-A-Service-Backlog, 10rsaas-editquality: Extend user group features - https://phabricator.wikimedia.org/T143909#2584281 (10Halfak) Gotcha. Thanks. Fixing. [22:27:58] Hmm... Looks like we didn't get much more fitness for the ruwiki models [22:31:02] 10Revision-Scoring-As-A-Service-Backlog, 06Collaboration-Team-Triage, 10Edit-Review-Improvements, 10MediaWiki-extensions-ORES: Include goodfaith model information in ORES review tool - https://phabricator.wikimedia.org/T137966#2584430 (10jmatazzoni) We're looking into this question pretty thoroughly as par... [22:38:51] 10Revision-Scoring-As-A-Service-Backlog, 06Collaboration-Team-Triage, 10Edit-Review-Improvements, 10MediaWiki-extensions-ORES: Include goodfaith model information in ORES review tool - https://phabricator.wikimedia.org/T137966#2584443 (10Halfak) @jmatazzoni agreed on the overlap. Though, I wonder if it wo... [23:45:19] 06Revision-Scoring-As-A-Service, 10Beta-Cluster-Infrastructure: deployment-sca03 can't call puppetmaster - https://phabricator.wikimedia.org/T143958#2584624 (10Ladsgroup)