[05:22:57] (03PS2) 10Ashuro07: Upgrade tests to WebdriverIO v-5 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/590259 (https://phabricator.wikimedia.org/T248223) [05:48:52] (03PS3) 10Ashuro07: Upgrade tests to WebdriverIO v-5 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/590259 (https://phabricator.wikimedia.org/T248223) [10:06:37] (03PS1) 10Vidhi-Mody: Upgraded WebdriverIO from v4 to v5 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/591791 (https://phabricator.wikimedia.org/T250900) [10:33:28] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Patch-For-Review: Upgrade WebdriverIO version to v5 in extensions/ORES - https://phabricator.wikimedia.org/T250900 (10Aklapper) @Vidhi-Mody: Assuming this task is about the #MediaWiki-extensions-ORES code project, hence adding that project tag so other... [12:15:38] 10MediaWiki-extensions-ORES, 10Scoring-platform-team, 10Patch-For-Review: Upgrade WebdriverIO version to v5 in extensions/ORES - https://phabricator.wikimedia.org/T250900 (10Vidhi-Mody) >>! In T250900#6078049, @Aklapper wrote: > @Vidhi-Mody: Assuming this task is about the #MediaWiki-extensions-ORES code pro... [13:04:28] Hey folks! I'm around for a couple hours today. [13:30:07] chtnnh, what do you think of trying to put together some code to add the draftquality predictions to people's wiki accounts? [13:30:35] sure i would love to help move the review along halfak [13:30:54] would need some pointers to get started though i feare [13:31:08] *fear [13:31:10] Oh! Actually, I think we should belay that. Let Helder and GoEThe work help us with evaluating draft quality and let's try to get some updates to the model done. [13:31:23] We have a better query and some features to add :) [13:31:25] you mean articlequality? [13:31:54] Nope. [13:31:55] https://quarry.wmflabs.org/query/43197 [13:31:59] yeah i am currently working on that [13:32:06] But I guess we can add the words_to_watch feature to articlequality too :) [13:32:15] yeah similar to enwiki [13:32:39] i was wondering how we could help ptwikipedians see the draftquality model performance? [13:32:57] Oh yeah! Looks like we can boost our observation count to 1450 per class. [13:33:03] We catch a lot more spam with this new query [13:33:27] 1481 spam (up from ~700), 1591 unsuitable (Up from ~1300 I think) [13:33:42] More data will help the model work better and deal with nuance better. [13:36:17] yes i agree [13:36:30] let me change those numbers in the makefile and update the query url [13:39:20] do you want me to change the name of balanced_3k to balanced _5k? [13:40:08] halfak ^ [13:40:16] hi! [13:40:21] halfak, speaking of features, do you know why [13:40:21] https://ores.wikimedia.org/v3/scores/ptwiki/57185234/articlequality?features [13:40:21] does not show "ptwiki.revision.non_cite_templates", which is present at [13:40:21] https://github.com/wikimedia/articlequality/blob/master/articlequality/feature_lists/ptwiki.py#L61-L64 [13:40:21] ? [13:40:39] hi helder [13:41:10] Helder, yes! That's a "meta feature" so we simplify it to it's basic components -- the count of cite templates and the count of non-cite templates. [13:41:18] *the count of all templates [13:41:41] Sorry that was confusing. But essentially by simplifying these it makes it a bit easier to modify them [13:42:06] E.g. if we have feature C that is A - B, we'll just show A and B because C is a simple function of A and B. [13:42:39] hum, ok [13:49:00] This actually happens with a lot of features. E.g. if we have something that looks like A/max(B, 1), it's really convenient to just have A and B be variables. This is a common pattern where we have something like cite_template_proportion = cite_templates / max(templates, 1) [13:49:23] The max(..., 1) is prevent division by zero :D [13:49:33] got it! [13:50:06] changing subject: could I suggest that when queries like [13:50:07] https://github.com/wikimedia/draftquality/blob/f2ca78313b49ae31b2a0e64baa2597a9f05c460a/Makefile#L267 [13:50:27] are mentioned in the code, that you also add the link to the query code as a comment? [13:51:03] It is not easy to find the "query" number from the "run" number [13:51:14] Right. We usually do that. I'd just forgot to insist on it this time. [13:51:35] I should submit a PR to quarry to make turning a run # into a query # easier [13:51:51] But there's a lot of things I *should* do :) [13:51:58] chtnnh, ^ [13:52:02] I can imagine [13:52:15] Can you add the query link as a comment to the Makefile when you grab the new data? [13:52:16] let me add that up real quick [13:52:22] Thank you! [13:55:00] halfak, is https://github.com/wiki-ai dead? [13:55:08] were things moved to wikimedia? [13:55:13] same question [13:55:14] Practically dead, yes. [13:55:25] We moved all of the repos related to production code over to wikimedia/ [13:55:43] right [13:55:46] Every now and then I go revive a repo that was sitting in wiki-ai. [13:56:15] do you see an alert when you open https://github.com/wiki-ai/jade ? [13:59:16] chtnnh, did you see my comments at https://github.com/wikimedia/draftquality/commit/affed3c3fc398058c1b4c6018a98782e93331405#diff-b67911656ef5d18c4ae36cb6741b7965R271 [13:59:18] ? [14:00:31] yes i did actually [14:00:38] just forgot to reply [14:00:40] on it [14:00:50] ah, ok [14:05:38] replied Helder :D [14:19:06] chtnnh, thanks! I left two more comments there :) [14:19:35] just gonna check them out once i am able to build the new draftquality model [14:19:48] that is fine [14:21:56] (03PS2) 10Vidhi-Mody: Upgraded WebdriverIO from v4 to v5 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/591791 (https://phabricator.wikimedia.org/T250900) [14:49:31] replied Helder [14:51:34] That second shuf is for re-shuffling the whole set :) [14:53:44] basically [16:02:26] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10He7d3r) How should we interpret the different weighted sum values (shown in the parenthesis) for articles such as https://pt.wikip... [16:11:23] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) That could be one way to think of it, yes. This is because the prediction is the label with the highest probability wherea... [16:20:16] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10GoEThe) >>! In T250809#6076864, @GoEThe wrote: > That one was evaluated in 2008, so standards probably have changed. Looking at... [16:21:41] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) That seems to make the models prediction right, is that correct @GoEThe ? [16:22:24] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10GoEThe) >>! In T250809#6078443, @Chtnnh wrote: > That seems to make the models prediction right, is that correct @GoEThe ? I woul... [16:25:02] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) The model seems to be working with an accuracy of 80% from the numbers @He7d3r has reported. I think this review could ben... [16:31:24] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10GoEThe) Yes, that's a good idea. You can post this message on the Esplanada, our version of the Village Pump (https://pt.wikipedia... [16:35:15] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) Whichever is a easiest. I would personally prefer that someone could translate it to Portuguese first though. Maybe you ca... [16:37:49] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Halfak) The "weighted sum" is essentially the center of the probability distribution across classes. E.g. Article A: ` 1: 10% 2... [16:45:59] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10GoEThe) >>! In T250809#6078469, @Chtnnh wrote: > Whichever is easiest. I would personally prefer that someone could translate it t... [16:53:07] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) The script is all credit to @Halfak 😄 Thank you for the translation. I will add my message on Esplanada now! [16:57:53] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10He7d3r) >>! In T250809#6078470, @Halfak wrote: > (...) > When we don't, that probably indicates that there are many articles with... [17:01:01] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) @GoEThe Added my message on Esplanada, do check it out and correct it if anything seems wrong about it [17:04:37] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10GoEThe) I would just paste the content of the etherpad on Esplanada. And don't forget to finish you message with four tildes ("~~~... [18:04:34] 10ORES, 10Scoring-platform-team, 10artificial-intelligence: Review model performance for ptwiki 'articlequality' and 'draftquality' - https://phabricator.wikimedia.org/T250809 (10Chtnnh) @He7d3r has updated the message. What do you think about it now? I think we should see some input from the community on th... [18:22:37] chtnnh, check out this very early draft: https://meta.wikimedia.org/w/index.php?title=User:He7d3r/Tools/DraftAndArticleQuality.js&action=history [18:30:30] chtnnh, for now it does only this: https://postimg.cc/vgL61KQ6 [18:30:56] notice the new [OK] and [unsuitable] texts on the left of each new page [18:32:48] chtnnh, I loaded the code above using this: http://pt.wikipedia.org/wiki/?diff=58098992 [18:39:54] let me just have a look helder [19:02:43] the script seems like a great initial step helder [19:03:14] would it be possible to keep the draftquality and articlequality scripts separate tho? [19:03:26] that would enable community members [19:04:17] to choose what functionality they wish to keep and would also make debugging the scripts easier [19:08:40] chtnnh, yes, however I started like this because I wanted to reuse the interface which was already available to interact with ores [19:09:13] oh ofcourse that makes sense too, i wonder what the best route is [19:09:25] re: keeping the files together or separate [19:10:24] in the long run, probably separate, with some common code in a third script which would be loaded as a dependency [19:11:01] For today, I'll leave it as is [19:11:12] sure it looks good this way too [19:12:00] OresScoreBatcherPool and OresApi are the parts which would be common, I think [19:13:29] i need to brush up on my js before i can comment on that XD [19:14:42] I'll have to go now