[09:47:57] (03PS1) 10Ladsgroup: Introduce ORESServices and use it [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441809 [09:48:42] (03CR) 10Ladsgroup: Introduce ORESServices and use it (031 comment) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441809 (owner: 10Ladsgroup) [09:50:52] (03CR) 10jerkins-bot: [V: 04-1] Introduce ORESServices and use it [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441809 (owner: 10Ladsgroup) [09:52:02] (03PS2) 10Ladsgroup: Introduce ORESServices and use it [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441809 [10:20:32] (03CR) 10Awight: [C: 032] "Fancy!" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441809 (owner: 10Ladsgroup) [10:25:15] (03Merged) 10jenkins-bot: Introduce ORESServices and use it [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441809 (owner: 10Ladsgroup) [10:26:59] (03CR) 10jenkins-bot: Introduce ORESServices and use it [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441809 (owner: 10Ladsgroup) [10:30:36] 10Scoring-platform-team (Current), 10ORES, 10Wikidata, 10User-Ladsgroup: new ORES labeling campaign for Wikidata - https://phabricator.wikimedia.org/T195701#4310952 (10Ladsgroup) Last time we did it with 500K. I think that's enough [10:41:25] 10Scoring-platform-team, 10Wikilabels: Labeling campaign for dewiki shows 'revision not found' for all revisions - https://phabricator.wikimedia.org/T194457#4310991 (10Ladsgroup) That is indeed definitely a problem I can't reproduce. My guess is something wrong with your CORS settings, Can you get me a screens... [11:40:14] I'm starting to believe this note about new page patrol: [11:40:16] > No two patrollers seem to utilize the same process. [12:06:16] 10Scoring-platform-team, 10JADE: Integrate JADE with MediaWiki "patrol" action - https://phabricator.wikimedia.org/T198085#4311419 (10awight) [12:07:40] 10Scoring-platform-team, 10JADE: Integrate JADE with PageTriage "Mark as reviewed" action - https://phabricator.wikimedia.org/T198086#4311433 (10awight) [12:30:33] 10Scoring-platform-team, 10JADE: Integrate JADE with FlaggedRevs manual review actions - https://phabricator.wikimedia.org/T198090#4311525 (10awight) [12:39:15] 10Scoring-platform-team (Current), 10JADE, 10Operations, 10User-Joe: Extension:JADE scalability concerns due to creating a page per revision - https://phabricator.wikimedia.org/T196547#4311554 (10awight) I'm looking at two more data sources that we may decide to integrate with: PageTriage and FlaggedRevs.... [12:51:48] (03PS1) 10Ladsgroup: Graceful handle of exceptions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441856 (https://phabricator.wikimedia.org/T197647) [12:54:49] (03CR) 10jerkins-bot: [V: 04-1] Graceful handle of exceptions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441856 (https://phabricator.wikimedia.org/T197647) (owner: 10Ladsgroup) [12:55:00] (03CR) 10jerkins-bot: [V: 04-1] Graceful handle of exceptions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441856 (https://phabricator.wikimedia.org/T197647) (owner: 10Ladsgroup) [13:04:37] 10Scoring-platform-team, 10MediaWiki-Change-tagging: "patrol" API endpoint "tags" parameter is unused - https://phabricator.wikimedia.org/T198094#4311628 (10awight) [13:06:10] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10Patch-For-Review, 10User-Ladsgroup: Gracefully handle edge case where wiki is not receiving edits - https://phabricator.wikimedia.org/T197647#4311652 (10Ladsgroup) a:03Ladsgroup [13:10:10] 10Scoring-platform-team (Current), 10ORES, 10editquality-modeling, 10artificial-intelligence: Duplicated feature name in editquality - https://phabricator.wikimedia.org/T197679#4311666 (10awight) The next step is to rebuild all editquality models... [13:23:04] back in an hour... [13:50:47] 10Scoring-platform-team, 10MediaWiki-Change-tagging: "patrol" API endpoint "tags" parameter is unused - https://phabricator.wikimedia.org/T198094#4311869 (10TTO) All logged actions are supposed to allow tagging. Not many people have taken advantage of this feature for various reasons, and I would imagine uptak... [14:28:47] 10Scoring-platform-team, 10MediaWiki-Change-tagging: "patrol" API endpoint "tags" parameter is unused - https://phabricator.wikimedia.org/T198094#4312095 (10awight) @TTO Makes sense to me... Maybe part of the issue is that the documentation implies that we should be tagging actions with something like a user-... [14:46:25] 10Scoring-platform-team, 10Huggle, 10JADE: Use JADE as a repository for ORES counterexamples - https://phabricator.wikimedia.org/T197098#4312166 (10awight) I found some references to the `action=patrol` API in Huggle source, but didn't realize until now that it isn't used in practice: ``` SELECT params["ac... [14:51:44] 10Scoring-platform-team, 10MediaWiki-Change-tagging: "patrol" API endpoint "tags" parameter is unused - https://phabricator.wikimedia.org/T198094#4312193 (10awight) p:05Triage>03Lowest Turning this into a minor cleanup thing... [14:52:43] 10Scoring-platform-team, 10MediaWiki-Change-tagging: "patrol" API endpoint "tags" parameter is unused - https://phabricator.wikimedia.org/T198094#4312196 (10awight) 05Open>03Invalid Actually, throwing it out. [14:58:39] Hey folks [14:58:43] RUnning a bit late to sync [14:58:50] Plz start on current_work without me. [14:58:58] will do [15:05:24] halfak: ping [15:53:32] halfak: want to chat about the JADE schema for a few minutes? Or async: https://phabricator.wikimedia.org/T179301#4308825 [15:53:49] I've got time for sync :) [15:54:00] cool [15:54:14] I do like the edit comment/change tags strategy [15:54:28] The current common practice for tool edits is to append a link [15:54:41] e.g. Huggle's is "([[WP:HG|HG]])" [15:54:42] 10[1] 04https://meta.wikimedia.org/wiki/WP:HG [15:54:46] ah, that works. [15:54:47] Thanks AsimovBot [15:55:18] awight, I don't like it as much, but I agree that the edit-level is the right place to have an "origin" tag. [15:55:44] I wish that "revision" had a field for "origin" [15:55:46] So that would take care of "origin". IMO that's plenty for analysis, yeah it doesn't give a completely clean signal, but should be pretty good assuming that tools do the bulk of editing. [15:55:50] +1 [15:56:04] Oh! There's also consumer ID tags in change tags. [15:56:11] "user name" and "guid" are in revision, so I'd like to drop that too [15:56:17] yeah change tags would work for me. [15:57:00] Na. guid is like a signature on a talk page. [15:57:11] sorry, I think threads crossed. [15:57:13] Revision is not enough [15:57:26] ok I'm following you. [15:57:48] [I see, I'm the one who mixed topics] [15:58:27] Including guid has the nice property of making it possible to kludge endorsements-like behavior out of multiple judgments [15:59:05] has the bad property of making judgments seem personal, but that is the wiki reality [16:00:46] ah--I had some better arguments for dropping guid. guid never needs to be changed, so it doesn't need to be part of the editable page content. Editors can easily review the JADE page history to see what each user has changed. [16:00:47] Right. [16:01:06] awight, then why do people put signatures on talk pages? [16:01:29] That's when you have a threaded discussion, which this is not [16:01:31] I don't think "easy" and "review page history" go together. [16:01:36] they don't sign article content, for example [16:01:41] this isn't a talk page... [16:01:45] ^ lol [16:02:06] Right. I think endorsements are discussion points and need a signature. But I don't think that judgements need guids. [16:02:09] yeah I had qualms as I was typing that. lack of a "blame" feature is insane. [16:02:14] aha okay [16:02:27] well I'm arguing for dropping endorsements as well [16:02:30] for now. [16:05:20] w/o guid in judgments, it would be silly to create multiple redundant judgments to emulate endorsements. So +1s would have to be made on the talk page. [16:07:53] So... contradicting myself, it would also make sense to me if we include judgment.comment, in the content and if we do, then judgment.guid is sane. [16:08:09] that gives us structured endorsement if we want to kludge them for now. [16:08:30] It also lets reviewers edit the justification, which seems important. [16:10:03] I think judgement.comment without judgement.guid is sane. [16:10:25] I think iterating on a good explanation for the judgement is a good collaborative behavior. [16:10:42] OK that would be cool, in that case let's call it "judgment.justification" though? [16:10:43] I'm also OK with keeping endorsements to the talk page for now. Doesn't seem too crazy and would allow us to explore more. [16:11:00] I think comment is better because, maybe it's a summary and not a justification? [16:11:07] comment is more general of a term. [16:11:38] to my ear, "comment" implies something less collaborative [16:11:45] "notes"? [16:12:21] I like the idea of making it more general, as you said. [16:14:34] revision uses "comment" [16:14:39] So does the log for admin actions. [16:14:46] But I get what you're saying. [16:14:51] In those cases, they are not collaborative [16:14:55] exactly [16:15:02] ===1 person [16:15:09] and people will be familiar with that term [16:15:34] I don't think "notes" is awesome, but it seems to describe what we're looking for. [16:22:02] * awight brings down the gavel [16:22:12] kk I can work with this for now [16:23:41] "annotation"? [16:23:57] Maybe even "tags" [16:24:31] Seems like we want a mix. [16:24:38] humm. annotation is a fancy word for notes eh. What do you mean by a mix? [16:24:48] "Trying to update a data item #data #mistake #nocite" [16:25:10] interesting [16:25:58] Something else to consider I suppose--what if users start having conversations in this field. [16:26:18] I guess we learn, no harm done. [16:28:11] Here's a random thought I had: this technically creates a new form of work for community members. Will imparting JADE judgments be seen as a backlog that people will need to go through? Will this incentivize shortcuts? [16:28:16] halfak: Did you want me to be at the SP<->Research meeting for any reason? I would relocate otherwise. [16:28:29] harej: I do worry about that. [16:28:33] I'm not sure. Never had these meetings before. [16:29:08] harej: The way we're harvesting the data transparently from existing workflows means that the work of writing judgments shouldn't increase for anyone. [16:29:27] I know we don't want every edit to go through JADE, but is that something that might be theoretically desirable? Might people want to at least go down that road? [16:29:28] However, they will appear as recentchanges, so there's a load of meta-patrol. [16:29:59] harej: It's not possible to do with human labor alone. [16:30:38] We've talked about JADE serving as a general storage for human- or machine-produced judgments, but for now we have to target only human judgments cos of scalability. [16:31:26] Which leads to technical shortcuts. A very common Wikipedia thing is to graft an auto-workflow on top of a manual workflow. But in this case, the fact of it being a manual workflow is very intentional. But that's not going to stop people from thinking "we need to annotate all of the revisions," or "all of the revisions of class X," and grafting auto-judgments on top of a system designed for human judgments. [16:33:31] harej: +1, I've done some estimates though and it seems that with the workflows I'm targeting it's possible to discriminate between manual and semi-automatic [16:33:44] e.g. FlaggedRevs approve-a vs approve [16:34:21] So, scalability concerns aside, we don't really care if people end up doing auto-judgments because we can tell? [16:34:36] mmm... [16:34:47] um not if they add new workflows which fool us :) [16:34:57] e.g. something hitting the action=patrol API [16:35:56] * halfak --> Meeting [16:38:05] harej: sorry, so generally we do care and we're actively discouraging even semi-auto judgments [16:38:30] I'm interested in exploring whether we'll be creating a shortcut disaster waiting to happen. [16:39:58] One problem I see is that I don't have any other criteria to suggest for which revisions should be judged, other than "we only want humans" [16:41:40] I don't think we care whether they're skewed towards "damaging" or "not damaging", nor whether it's exclusively used for ORES false positive reports, or for everything being patrolled generally. [16:42:52] (03PS2) 10Ladsgroup: Graceful handle of exceptions [extensions/ORES] - 10https://gerrit.wikimedia.org/r/441856 (https://phabricator.wikimedia.org/T197647) [16:47:28] I think as long as we don't present reviewing as a "backlog" then we should avoid a lot of the FUD that exists around backlogs. [16:47:56] Honestly I feel like a lot of the Wikipedia editor user experience feeds into backlog anxiety by just overwhelming you with all the stuff there is to do. [16:48:14] With no clear way to get things done or even mark things as done, which makes the problem worse over time... [16:53:51] LOL [16:58:07] Yes I think we'll be able to avoid creating another backlog, and might have some success in reducing existing backlogs if we can encourage more data sharing between existing tools. [16:58:24] However, we'll create a new namespace and many new edits that might be considered patrolling targets. [16:58:40] That seems like a tangible social issue that you will be entangled in soon enough. [17:05:52] harej: ^ [17:05:58] gtg for now [17:08:58] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4312705 (10awight) Mini-consensus in IRC was to have judgments with only "data" and "notes" fields, and drop endorsements for now. We'll encourage collaborative revision o... [17:33:12] Re. backlogs, I agree with awight. If anything, we're building a tool to help make a few types of backlogs easier to deal with. [17:33:26] The new work will really come with any sort of purposeful audit of ORES. [17:33:45] Which may or may not be in the interests of any subset of our users. [18:07:44] So what is the likelihood that we migrate JADE to MCR? [18:12:04] Short term, 0% [18:12:10] Long term, who knows? [18:12:26] I think MCR will need to abide by Risker's checklist before we'll consider it. [18:17:44] Also I am eager to not sign us up for platform-level experiments like that. [18:41:21] 10Scoring-platform-team (Current), 10ORES, 10Wikidata, 10User-Ladsgroup: new ORES labeling campaign for Wikidata - https://phabricator.wikimedia.org/T195701#4313015 (10Halfak) I think that should be the plan then. Query for a random sample of 500k. Then select *needs_review* from that set. [18:44:11] awight, hey dude. [18:44:26] I was just reviewing our goals for this quarter and it looks like the ORES-JADE connector was on there [18:44:39] It looks like we're not going to finish it after all [18:45:01] I'm thinking that I'll report that our work with designers on JADE took higher priority so we switched it out. [18:45:03] Sound OK? [19:01:15] * halfak is running away to bring his doggo to the vet [19:34:50] halAFK: For sure—we made architectural changes that completely changed how the ORES-JADE connector would function, so it got put on ice until we could solidify the JADE core... [20:36:13] 10Scoring-platform-team (Current), 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: Fix "score_schema" -- invalid JSON Schema - https://phabricator.wikimedia.org/T197828#4313332 (10Halfak) https://github.com/wiki-ai/revscoring/pull/404 [20:36:22] 10Scoring-platform-team (Current), 10Analytics, 10Analytics-Kanban, 10EventBus, and 3 others: Fix "score_schema" -- invalid JSON Schema - https://phabricator.wikimedia.org/T197828#4313333 (10Halfak) a:05Ottomata>03Halfak [20:39:53] wiki-ai/revscoring#1497 (schema_fix - befc885 : halfak): The build passed. https://travis-ci.org/wiki-ai/revscoring/builds/396583198 [20:51:49] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366#4313349 (10Halfak) @Liuxinyu970226, I'm trying to work out whether our informals detection strategy is just not going to work or if there's just some inf... [20:53:09] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Add wp10 model in PetScan - https://phabricator.wikimedia.org/T197871#4313355 (10Halfak) Who maintains petscan? It seems like we should get them involved. We're working on bringing wp10 predictions to MediaWiki, so if petscan us... [21:06:06] 10Scoring-platform-team, 10Huggle, 10JADE: Use JADE as a repository for ORES counterexamples - https://phabricator.wikimedia.org/T197098#4313363 (10Petrb) @awight that's because you are probably searching only English wikipedia where patrolling was disabled long time ago, here is very old note from config pa... [21:06:06] 10Scoring-platform-team, 10Huggle, 10JADE: Use JADE as a repository for ORES counterexamples - https://phabricator.wikimedia.org/T197098#4313364 (10Petrb) The change was made on 2005 so I don't know if it's still valid :P [21:16:30] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Add wp10 model in PetScan - https://phabricator.wikimedia.org/T197871#4313395 (10Theklan) I think @Magnus is involved. [21:24:58] 10Scoring-platform-team, 10Wikilabels, 10artificial-intelligence: Implement a modeling self-check process - https://phabricator.wikimedia.org/T198144#4313407 (10Halfak) [21:26:19] 10Scoring-platform-team, 10Wikilabels, 10artificial-intelligence: Implement a modeling self-check process - https://phabricator.wikimedia.org/T198144#4313427 (10Halfak) @notconfusing, is this something you would be interested in looking into for your first project. I'm thinking that we follow the process ma... [21:28:27] 10Scoring-platform-team, 10Analytics, 10Analytics-Kanban, 10EventBus, and 4 others: Modify revision-score schema so that model probabilities won't conflict - https://phabricator.wikimedia.org/T197000#4313449 (10Halfak) [21:30:14] 10Scoring-platform-team (Current), 10Analytics, 10EventBus, 10ORES, and 3 others: Invalid field names in ORES models causing downstream Hive ingestion to fail - https://phabricator.wikimedia.org/T195979#4243147 (10Halfak) Well, the field names have nothing wrong with them. Essentially, "true" and "false"... [21:31:52] 10Scoring-platform-team (Current), 10ORES: Investigate what is creating Redis transactions and whether it can be fixed - https://phabricator.wikimedia.org/T196889#4313467 (10Halfak) Relevant: https://github.com/celery/celery/issues/3500 [21:32:11] 10Scoring-platform-team, 10ORES: Investigate what is creating Redis transactions and whether it can be fixed - https://phabricator.wikimedia.org/T196889#4313469 (10Halfak) [21:37:10] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4313509 (10Halfak) +1 [21:38:13] 10Scoring-platform-team (Current), 10articlequality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Article quality campaign for Persian Wikipedia - https://phabricator.wikimedia.org/T174684#4313511 (10Halfak) 45% done! @Ladsgroup, do you think you could give this another push? [21:38:57] 10Scoring-platform-team, 10articlequality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Article quality campaign for Persian Wikipedia - https://phabricator.wikimedia.org/T174684#4313516 (10Halfak) [21:39:05] 10Scoring-platform-team, 10articlequality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Train and test wp10 model for fawiki - https://phabricator.wikimedia.org/T190050#4313517 (10Halfak) [21:41:54] 10Scoring-platform-team (Current), 10Wikimedia-Hackathon-2018, 10User-Ladsgroup: Audit deployed editquality models and figure out why if the models are bad - https://phabricator.wikimedia.org/T194742#4313524 (10Halfak) @Zache, @eranroz, and @Wargo, how are ORES damage detection models working on your wikis?... [21:43:25] 10Scoring-platform-team (Current), 10Wikimedia-Hackathon-2018, 10User-Ladsgroup: Improve features for wikibase vandalism detection model - https://phabricator.wikimedia.org/T194737#4207098 (10Halfak) Any updates here? [21:43:31] 10Scoring-platform-team (Current), 10Wikimedia-Hackathon-2018, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Improve features for wikibase vandalism detection model - https://phabricator.wikimedia.org/T194737#4313538 (10Halfak) [21:43:54] 10Scoring-platform-team, 10edittypes-modeling, 10artificial-intelligence: Train edit types model on labeled data for English Wikipedia - https://phabricator.wikimedia.org/T121715#4313543 (10Halfak) a:05Sumit>03None [21:44:07] 10Scoring-platform-team, 10ORES, 10edittypes-modeling, 10artificial-intelligence: Train and test edit type model for Catalan Wikipedia - https://phabricator.wikimedia.org/T186749#4313546 (10Halfak) [21:56:24] 10Scoring-platform-team, 10Wikilabels, 10artificial-intelligence: Implement a modeling self-check process - https://phabricator.wikimedia.org/T198144#4313407 (10Maximilianklein) @Halfak , this is a good idea, I would be glad to take it on. Another sort of crazy idea is we can build a test-set for mislabelled...