[13:42:57] 10Jade, 10Scoring-platform-team (Current): Implement secondary schemas for joining Jade data to other tables - https://phabricator.wikimedia.org/T229977 (10Halfak) This looks related: https://www.mediawiki.org/wiki/Extension:MachineVision/Schema [13:52:11] 10Jade, 10Scoring-platform-team (Current): Implement secondary schemas for joining Jade data to other tables - https://phabricator.wikimedia.org/T229977 (10Halfak) Also possibly relevant is: https://mariadb.com/kb/en/set-data-type/ So, it looks like these set types are essentially implemented as bitmasks. I'... [14:24:43] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) [14:44:23] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Halfak) [14:44:25] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10artificial-intelligence: Build draft quality model for ptwikipedia - https://phabricator.wikimedia.org/T246667 (10Halfak) [14:44:27] 10Scoring-platform-team (Current), 10Wikilabels, 10articlequality-modeling, 10artificial-intelligence: Build article quality model for ptwikipedia - https://phabricator.wikimedia.org/T246663 (10Halfak) [14:46:02] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Halfak) Right now, we don't have a good way to surface `draftquality`. We don't have to do anything special. We could just add a... [14:54:12] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10He7d3r) Maybe only display the draftquality if the article was created no more than X days ago? Or has no more tha Y revisions? I... [15:00:19] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Halfak) That sounds totally reasonable to me. I don't think there is any such flag, but I'd support any definition of "new" that... [15:02:08] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10GoEThe) Special:NewPages only lists articles under 30 days old. I would go with that. [15:04:13] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10He7d3r) If I'm not mistaken, we can get the earliest revision of the article using something like this: https://pt.wikipedia.org/w... [15:04:50] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) I think @GoEThe 's suggestion makes a lot of sense. That would cover a reasonable amount of articles and would still be re... [15:11:22] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Halfak) We could just make the predictions show up on https://pt.wikipedia.org/wiki/Especial:P%C3%A1ginas_novas [15:15:47] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) That would be very handy, it would make the review process much faster for the ptwikipedians. [15:23:35] 10Scoring-platform-team, 10VisualEditor, 10edittypes-modeling, 10Editing-team (Q3 2019-2020 Kanban Board): Change from source code editing to visual editing: minor edit property not kept - https://phabricator.wikimedia.org/T250388 (10JTannerWMF) We will take a look at this [15:30:07] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10He7d3r) Also, it would be interesting to generate a list of articles where ORES quality prediction differs from the current automa... [15:49:39] Hmm. We could at least do a randomly sampled set for the difference in prediction. [15:50:27] We already have text associated with labels in our datasets. chtnnh what do you think about scoring a bunch of these versions and generating a sample of disagreeing predictions? [15:50:53] We can use the "revscoring score_revisions" utility to add a prediction to labeled dataset. [15:53:07] Looks like github is struggling at the moment. [15:53:09] chtnnh, ^ [16:05:53] chtnnh, https://pastebin.com/0dkM4TsN [16:07:49] yeah i think thats a promising idea [16:08:06] re: generating sample of disagreeing predictions [16:08:31] how do you propose we utilize this feedback though? [16:09:11] halfak ^ [16:17:29] I think we post it on a wiki page. Let ptwiki editors read through it and talk to us about where the model is right/wrong. [16:17:47] Generally, we'll see patterns emerge that will give us clues about improvements we can make. [16:33:32] halfak, also, maybe even the "confusion matrix" of "ores predictions" against "lua module predictions" could reveal something which we should look at more carefully [16:33:55] which classes differ the most [16:42:16] sounds like a good idea halfak [16:42:26] re: posting it on wiki pages [17:15:26] kevinbazira: Looks like we sent our updates at the same time [17:15:38] :-D [17:15:46] Wooops! [17:16:04] I'll resend mine to your thread [17:16:14] I wonder how such coincidences happen [17:16:58] Sorry about that :) [17:34:38] haksoat, any notes on performance from the ES tokenizer? [17:51:04] halfak|Lunch: Not yet [17:51:47] Not able to extract named groups so far [17:52:22] Also I'll need some help with how to go about profiling the performance [17:52:31] halfak|Lunch: [19:04:55] haksoat, I can help with performance :) [19:05:08] Let me know if you want to have a chat. We could also cover this in our 1:1 tomorrow. [19:12:15] 10Scoring-platform-team, 10Continuous-Integration-Config: CI should check to see if our wheels are good - https://phabricator.wikimedia.org/T250746 (10hashar) [19:37:49] Okay. Great. [19:38:10] At the moment I'm just trying to compare the wikitext_split to elasticsearch [19:38:41] Also tried asking on stackoverflow and the elasticsearch forum if there's a way to extract the named groups [19:57:50] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10He7d3r) Out of the 951 featured articles (quality 6) on ptwiki: * 3 (0,3%) are predicted as having quality 3 * 37 (3,9%) are predi... [20:12:15] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10He7d3r) Out of the 974 good articles (quality 5) on ptwiki: * 12 (1,2%) are predicted as having quality 3 * 32 (3,3%) are predicte... [20:18:52] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Halfak) That sounds pretty decent. I looked at https://pt.wikipedia.org/wiki/Benjamin_Abrah%C3%A3o_Botto -- one of the articles t... [21:01:34] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10GoEThe) That one was evaluated in 2008, so standards probably have changed. [21:06:31] halfak, I tried setting the value of a feature using the url https://ores.wikimedia.org/v3/scores/ptwiki/57185234/articlequality?features&feature.ptwiki.revision.paragraphs_without_refs_total_length=500 [21:06:42] but it didn't accept the value 500, only 500.0 [21:06:52] shouldn't it convert integers to floats whenever it is possible? [21:13:45] (03PS1) 10Halfak: Adds working update_wheels.py utility. [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/591478 [21:14:17] Helder, indeed that's a limitation. We don't automatically handle compatible types. [21:14:34] And for lots of reasons it's useful to force everything to use floats. Sorry for the trouble! [21:14:44] accraze, I think you'll like this: https://gerrit.wikimedia.org/r/#/c/research/ores/wheels/+/591478 [21:14:58] I started writing the docs and instead decided to document via automation. [21:18:16] Helder, if you wanted to dig into handling that, I'd review that PR :) [21:29:41] (03CR) 10Accraze: [C: 03+2] Adds working update_wheels.py utility. [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/591478 (owner: 10Halfak) [21:29:50] nice one halfak [21:30:19] \o/ Hopefully this'll help cut down on human (aka Aaron) error [21:31:38] halfak, is that type compatibility supposed to be handled by revscoring? Or one of the other "layers" (articlequality, ores)? [21:33:06] revscoring handles feature types. [21:33:55] Helder, https://github.com/wikimedia/revscoring/blob/master/revscoring/features/feature.py#L99 [21:34:24] We might want to have some notion of "compatible" types. [21:47:55] Alright. I think I'm going to wrap up for the day. I'm working reduced hours the next three days. You'll find me around here between 1300 and 1500 UTC. [22:07:18] Hey halfak, do you know if the RecentChange table has ORES scores? [22:07:20] https://www.mediawiki.org/wiki/Manual:Recentchanges_table [22:14:01] xinbenlv, I think you will need this other table: https://github.com/wikimedia/mediawiki-extensions-ORES/blob/e421a8e118d392d153756368a13bbae84c3c22cb/sql/ores_classification.sql [22:15:03] Thanks Helder! [22:23:10] 10Jade, 10Scoring-platform-team (Current), 10Patch-For-Review: Implement secondary schemas for joining Jade data to other tables - https://phabricator.wikimedia.org/T229977 (10ACraze) Pushed up a WIP patch set that uses a basic ad hoc solution where the `jade_diff_label` has tinyint fields for `damaging` and... [22:23:52] Helder, that's awesome [22:23:55] thank you for answering it [22:24:27] you're welcome :) [22:34:08] 10Jade, 10Scoring-platform-team (Current), 10Epic, 10Patch-For-Review: Implement secondary Jade Integrations - https://phabricator.wikimedia.org/T229974 (10ACraze) Dropping my notes here on the current state of the secondary integrations. All of these hooks are currently disabled (minus MoveHooks) due to... [22:43:13] 10Scoring-platform-team, 10VisualEditor, 10edittypes-modeling, 10Editing-team (Q3 2019-2020 Kanban Board): Change from source code editing to visual editing: minor edit property not kept - https://phabricator.wikimedia.org/T250388 (10matmarex) I wrote a patch for this, was easy enough. But that only genera...