[00:08:07] 10JADE, 10Scoring-platform-team (Current), 10Discovery-Search, 10Elasticsearch: Extract judgment data for search indexing - https://phabricator.wikimedia.org/T206352 (10awight) Using the `cirrussearch` mw-vagrant role, I can confirm that the default search indexing isn't going to work well, we need to cust... [00:45:46] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Service hits for obtaining thresholds alert. [00:46:40] 10JADE, 10Scoring-platform-team (Current), 10Discovery-Search, 10Elasticsearch: Extract judgment data for search indexing - https://phabricator.wikimedia.org/T206352 (10awight) @harej: Just a heads-up that we can consider end-user use cases here, for example the ORES and JADE extensions could support an ad... [00:52:36] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [01:07:32] (03PS1) 10Awight: Streamline search results summary [extensions/JADE] - 10https://gerrit.wikimedia.org/r/469154 (https://phabricator.wikimedia.org/T206352) [01:13:48] 10JADE, 10Scoring-platform-team (Current), 10Discovery-Search, 10Elasticsearch, 10Patch-For-Review: Extract judgment data for search indexing - https://phabricator.wikimedia.org/T206352 (10awight) a:05awight>03None We might want to merge the minor change above, and circle back to the question of addi... [16:40:04] halAFK: srrodlund: I'm at an Ally Skills workshop, realizing now that it's booked over our docs meeting, sorry! [16:40:44] Oh that's okay I think halAFK may be at techconf [16:41:02] I'm swamped anyway, weeeeee! So I will move the meeting [16:41:05] any pref times [16:41:08] No worries. I'm conferencing anyway. [16:41:12] o/ [16:41:17] I have no preference, thanks [16:51:29] (03CR) 10Hoo man: [C: 032] Services tests (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468280 (owner: 10Awight) [16:53:59] (03CR) 10Awight: Services tests (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468280 (owner: 10Awight) [16:55:14] (03PS1) 10Awight: Minor fixups [extensions/JADE] - 10https://gerrit.wikimedia.org/r/469227 [17:01:37] (03Merged) 10jenkins-bot: Services tests [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468280 (owner: 10Awight) [17:08:27] (03CR) 10jenkins-bot: Services tests [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468280 (owner: 10Awight) [17:42:34] hi awight! [17:42:40] long time! [17:46:53] saurabhbatra: hey, how's it going :) [17:47:19] awight: doing okay :-) [17:47:33] finally have some time to work on the fraud thingy again [17:48:14] good time for you to talk? [17:49:26] saurabhbatra: I'm in a meeting for another hour, but after that, yes. [17:51:08] awight: ah, okay. I'll catch you here or in a mail thread sometime later today [17:52:29] That would be great, so nice to see you around! [18:59:14] (03CR) 10Hoo man: [C: 032] Tests for JudgmentTarget (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468277 (owner: 10Awight) [19:06:14] (03PS2) 10Awight: Minor fixups [extensions/JADE] - 10https://gerrit.wikimedia.org/r/469227 [19:06:24] (03CR) 10Awight: Tests for JudgmentTarget (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468277 (owner: 10Awight) [19:09:17] (03Merged) 10jenkins-bot: Tests for JudgmentTarget [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468277 (owner: 10Awight) [19:12:21] (03CR) 10jenkins-bot: Tests for JudgmentTarget [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468277 (owner: 10Awight) [19:12:27] (03CR) 10Hoo man: [C: 04-1] "Some style stuff, fine otherwise" (0310 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468284 (owner: 10Awight) [19:30:09] (03CR) 10Hoo man: [C: 032] Test fixes and coverage (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468625 (owner: 10Awight) [19:54:42] awight: talked to isaacj about finding revisions/diffs to put in front of people as part of JADE; ideas include people being able to nominate edits for further review (via JADE) and automatic recommendations for edits to review where raters disagree (e.g. one person says damaging another says non-damaging) [19:55:10] One of the questions that came up is if we want to recommend edits for review based on our "politics" or if we want to keep it as community-driven as possible. [19:57:02] The main challenge I think is making sure JADE is useful for more than just researchers [20:16:44] harej: Those sound like good ideas. [20:16:49] Another question is, how to recommend. [20:17:33] As in, the exact mechanism for surfacing recommendations, be it a special page or something else? [20:17:48] I've been thinking that yes we can make intepretations that guide which edits we ask people to review. There's a long-term goal of "disintermediation", where the machine learning system designers are no longer in charge of what happens, but it takes a lot of work to get there. [20:18:01] Yeah, it was the specific mechanism I was curious about. [20:18:42] Then, there are algorithmic ways to prioritize what to review, c.f. "active learning" [20:18:58] harej, don't forget suggesting judgments that our model predictions disagree with. [20:19:10] That was the first one I brought up :) [20:19:22] Aha. I didn't see it in your list. [20:19:30] Maybe that's because of my shoddy connection. [20:19:31] Er, I forgot to write it down [20:19:50] Model predictions disagree with... whom? [20:19:57] Humans [20:19:58] My thinking is we could surface ORES 50/50 cases [20:20:21] Or as close as 52-48 in either direction [20:20:29] Model says goodfaith. Human says badfaith. The model can learn the most from this example or it can help humans repair their judgment. [20:20:47] harej: https://www.mediawiki.org/wiki/JADE/Background#Actively_choosing_documents_to_reduce_labeling_burden [20:21:30] halAFK: how do we know what the human thinks before we ask them? [20:21:43] uh. We wouldn't harej [20:21:50] I'm thinking of a re-review process. [20:21:53] Just adding to the outcomes halAFK started listing, another possibility is that our judgment scale is poorly explained or poorly chosen. [20:21:54] Oh! [20:21:55] Who watches the watchment. [20:22:02] Meta-moderation [20:22:21] A re-review process makes sense. [20:28:54] (03CR) 10Hoo man: [C: 04-1] "Just found two minor things… still didn't manually test this." (032 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/466808 (https://phabricator.wikimedia.org/T202596) (owner: 10Awight) [20:36:10] another idea for selecting edits to review was those that are judged negatively by ORES but are from users who generally have a good history (or vice versa) [20:36:36] aha! Interesting. :) [20:36:46] Reputation vs. individual edit prediction. [20:47:40] yeah, I'll keep thinking on this too. really interesting set of questions / challenges [21:09:25] (03CR) 10Hoo man: "I have two little remarks, looks great otherwise :)" (032 comments) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/468180 (owner: 10Awight) [21:14:08] (03CR) 10Hoo man: [C: 032] "This is way better than the default (I presume) and a good starting point for further improvements." (031 comment) [extensions/JADE] - 10https://gerrit.wikimedia.org/r/469154 (https://phabricator.wikimedia.org/T206352) (owner: 10Awight) [21:14:27] (03CR) 10Hoo man: [C: 032] Minor fixups [extensions/JADE] - 10https://gerrit.wikimedia.org/r/469227 (owner: 10Awight) [21:14:49] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/JADE] - 10https://gerrit.wikimedia.org/r/469282 (owner: 10L10n-bot) [21:15:49] Hah, all reviewed :) [21:36:07] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/469290 (owner: 10L10n-bot) [22:11:06] O_O