[11:07:59] 10Scoring-platform-team, 10JADE, 10Design: JADE UI should provide an explanation of the target ORES score - https://phabricator.wikimedia.org/T181327#3786466 (10awight) [13:00:32] 10Scoring-platform-team, 10ORES, 10User-Zppix: Make ORES' webui when getting results easier to understand - https://phabricator.wikimedia.org/T181201#3786589 (10Ladsgroup) I wrote "at least" before that, it means I prefer a detailed look as well. What I was saying that it's blocked and should not be picked u... [13:02:01] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10Technical-Debt: Cleanup: Move processRevision out of Cache - https://phabricator.wikimedia.org/T181198#3782749 (10Ladsgroup) I think Cache itself needs to be rewritten and split into several lookup services. Hang on until it's done. [13:50:29] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10User-Ladsgroup: Split Cache.php to different services - https://phabricator.wikimedia.org/T181334#3786618 (10Ladsgroup) [13:53:43] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES: Redesign onRecentChange_save hook handler for ORES - https://phabricator.wikimedia.org/T181335#3786633 (10Ladsgroup) [13:53:51] 10Scoring-platform-team (Current), 10MediaWiki-extensions-ORES, 10User-Ladsgroup: Redesign onRecentChange_save hook handler for ORES - https://phabricator.wikimedia.org/T181335#3786646 (10Ladsgroup) a:03Ladsgroup [14:01:05] (03PS1) 10Ladsgroup: Warn instead of throwing error in case threshold is unparseable [extensions/ORES] - 10https://gerrit.wikimedia.org/r/393373 (https://phabricator.wikimedia.org/T181191) [15:40:28] o/ Amir1! [15:43:27] o/ codezee [15:52:59] o/ awight [15:53:04] Neat! [15:53:40] I read another fun paper & am typing some things into the [[mw:JADE/Background]] page... [15:53:41] 10[1] 10https://www.mediawiki.org/wiki/JADE/Background [15:54:05] Nice! I'm working on an old review now. Just got a rebuttal submitted so I need to pick through that. [15:54:12] I think I'll be pushing on JADE's API today. [15:54:42] Huh, all that sounds fancy [15:54:49] I sent some comments on the API PR, fwiw [15:56:11] There might be some more OAuth2 stuff we need to handle btw, but that should come later. [15:56:18] e.g. “refresh tokens" [15:57:45] Seems like maybe that'll be something we do in mwoauth and not JADE? [15:58:04] yes perfect [15:58:05] o/ [15:58:38] halfak: Do you happen to know why VE citations are so minimal on mw.org? [15:58:41] codezee: hi! [15:59:05] awight, no idea [15:59:28] Seems that the same extensions are available. [15:59:45] I’ve been creating citations on enwiki and porting the wikitext over... [16:09:11] awight: hi! [16:34:51] awight, I think I'll work on the "catch every exception" decorator idea I describe here: https://github.com/wiki-ai/jade/pull/4#discussion_r153049362 [16:35:10] Also, thanks for giving me an opportunity to rave about the def configure(..) pattern :) [16:35:14] It's so awesome :D [16:35:33] lol [16:35:36] I should write something about it :) [16:35:51] halfak: this is the fun thing I read, https://www.mediawiki.org/w/index.php?title=JADE/Background&diff=2628720&oldid=2623743 [16:35:55] “user co-training" [16:36:28] "there should be a positive weight on feature X" sounds like how huggle works [16:36:41] Except I think that's mostly based on petr's opinion of what weights should be [16:36:51] So, not really user-based. [16:37:17] Hmm, this study shows negative impact on classifier health, although they suggest it can be done in a better way and might be helpful. [16:47:30] It would miss pairs of features interacting in an “xor” way, in which varying one or the other would make the score move in an opposite direction as when both change together... [16:49:37] Right. I think that's why we use machine learning in the first place. [16:49:48] Such intricacies are exhausting. [16:53:26] haha oops I was chatting with apergos about “counterfactuals” in another window, mispasted [16:54:03] heh [16:54:09] backchannels ftw [16:54:13] Seems like it was relevant :) [16:54:57] So wrt. constraints, I was wooed by the “rationales” paper [16:55:24] But honestly don’t understand how the hell the rationales which are only applied to single documents can improve the overall classifer health when it hits new data. [16:55:37] Maybe I’ll selfishly ask others to read that paper to explain it to me. [16:59:57] I might give it a shot some time soon. [17:00:06] I'm trying my best to focus on making JADE work right now :D [17:00:15] We'll see if I get bored with that soon [17:01:28] halfak: give jade a diamond then maybe xD [17:02:07] * halfak starts considering JADE's retirement plan [17:02:09] brb [17:06:41] apergos: This is the paper about black-box auditing vs. laws: http://www-personal.umich.edu/~csandvig/research/Auditing%20Algorithms%20--%20Sandvig%20--%20ICA%202014%20Data%20and%20Discrimination%20Preconference.pdf [17:06:55] Apparently Sandvig has been teaming up with the ACLU to try to change the situation with CFAA [17:08:21] halfak: yeah a lot of what I’m reading will be in the distant future, not part of the JADE MVP. But I’m concerned about collecting a bunch of inferior data when we could be improving quality with a few basic tweaks to the interface. [17:08:27] maybe the rationales paper can be taken up in the informal research meeting [17:08:37] Cool idea! [17:08:53] It’s short, but mathy. [17:08:56] saved, thanks [17:09:06] yeah, i see that... [17:15:10] awight, not sure how the lit will affect data collection. [17:15:16] Any ideas on that front? [17:15:43] halfak: yes! glad you asked :p https://phabricator.wikimedia.org/T181327 [17:16:14] “sufficient low-level context” is the exciting part [17:16:40] I'm highly skeptical of this. I've never seen a user make use of an "explanation" in practice. [17:16:49] Also this has little to do with JADE. [17:17:05] and we should provide a way for the user to * flag features as “obviously” unhelpful, * flag features as correctly appropriate, and * add their own keywords we should look for [17:17:14] I think it’s very much to do with JADE [17:17:20] awight, also seems far out of scope for JADE [17:17:34] gotta pile out of the house ASAP but I’d love to chat more about that. [17:18:02] I really don't see how having users critique features (that they might not understand) will affect the quality of judgements [17:18:04] The lit points to us getting better quality feedback if we provide that explanation. Users begin to understand how the system "reasons" [17:18:23] happily, giving a space for free-form text feedback is shown to improve label quality [17:18:32] probably by making the end-user think harder [17:18:56] JADE isn't really a labeling UI. [17:19:00] Wikilabels, on the other hand... [17:19:13] “that they might not understand” is key, yeah. “sufficient” means, provide the minimal amount of low-level features, but enough. [17:19:27] Isn't it better to ask someone to make a judgement independent of ORES [17:19:40] I don't want ORES to affect people's judgements [17:19:49] We'll get deflated false positives/false negatives. [17:19:49] It’s not a labeling UI, but the data is garbage if the labels are wrong [17:20:04] awight, I don't think that's at stake here [17:20:21] One of the main points here is to have users critique the scoring engine, right? [17:20:46] Well, this will be a strong infrastructure for that and also training the prediction model too. [17:21:01] I agree in theory about not wanting to influence the humans’ judgment, but I don’t think we’re actually talking about a pure thing—we’re already coupled to ORES. [17:21:05] But it will also be an infra for inter-patroller communication. [17:21:25] JADE is bigger than ORES in that Wikilabels is outside of ORES. [17:21:27] That part is awesome and I haven’t found any literature covering the inter-human thing. [17:22:08] In our case, the labels will matter. I'm excited by the idea that Huggle could consume events from JADE to make sure multiple review is minimized -- or strategically applied. [17:22:19] petr has been asking for a better "patrolled" flag for a while. [17:22:29] JADE is like the ultimate "patrolled" flag. [17:22:54] +1 straying into active learning will ne fu [17:22:56] *be fun [17:23:18] In this case, I'm just thinking of direct-to-user usefulness [17:23:39] * apergos decides to up their ne fu [17:23:46] (sorry notsorry) [17:23:48] But yeah, active learning might have value, but I have some healthy skepticism about that technique. [17:23:59] o/ apergos :) [17:24:01] Nice, I think it’s productive that we chat about the motivations for JADE [17:24:13] apergos: ty that is a trademarked phrase though [17:24:13] just lurking, carry on! [17:24:16] wa [17:24:18] t [17:24:49] lol k I’m going to go ne fu, talk soon! [17:46:17] halfak: hey, I just got back from dinner [17:46:22] o/ [17:47:43] okay, what's up [17:47:57] I made a patch to make Stats handle things better [17:48:11] also made some tickets for the refactoring needed for the extension [17:54:07] Amir1, I saw that. I'm stoked you're picking that up./ [17:54:28] If you feel like looking at JADE stuff today, Adam picked up my PR and we started discussing. [17:54:34] * halfak gets link [17:54:52] https://github.com/wiki-ai/jade/pull/4 [17:55:25] I'm working on https://github.com/wiki-ai/jade/pull/4#discussion_r153043796 [17:55:27] coool [17:57:00] I will take some time to review and give notes [17:57:09] I might do it tomorrow or Monday [17:57:40] :) I' [17:57:49] ll have something new to critique soon. [18:45:27] 10Scoring-platform-team (Current), 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint-2017-11-22: ORES thresholds for Wikidata is too strict - https://phabricator.wikimedia.org/T180450#3758292 (10Halfak) Merged! Please make an update to our deploy config. [18:46:17] 10Scoring-platform-team (Current), 10JADE, 10Epic: Implement basic path structure for JADE (judgements) - https://phabricator.wikimedia.org/T181098#3786794 (10Halfak) Just made a big update based on some feedback from @awight. [19:02:39] Amir1, ^ new stuff to review [19:02:44] I'm heading out for the day [19:02:49] have a good one folks! [19:03:29] halfak: o/ [19:32:44] halfak: sorry for not doing more promptly but heres a fix to that one error that was due to users not having right version of python (IIRC it was the week before thanksgiving) [19:33:01] ^^ [19:34:09] wiki-ai/ores#844 (Pix1234-patch-1 - d0bb86c : Devin/Zppix): The build has errored. https://travis-ci.org/wiki-ai/ores/builds/307229129 [19:35:17] Amir1: is that error due to my pr? [19:40:03] Zppix: yes [19:40:12] Amir1: how so? [19:41:26] IDK [19:43:30] wiki-ai/ores#844 (Pix1234-patch-1 - d0bb86c : Devin/Zppix): The build has errored. https://travis-ci.org/wiki-ai/ores/builds/307229129 [19:46:45] All i did was add python_requires which WOULDNT affect the thing thats erroring [19:47:11] The app thats failing isnt in our repo either [19:47:20] Its on the virtenv itself [19:49:25] Amir1: its the nltk on stopworss [19:49:31] Stopwords? [20:30:54] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: Add accelerator keys (keyboard shortcuts) for forms in Wikilabels - https://phabricator.wikimedia.org/T162109#3786903 (10Aklapper) [20:32:58] 10Scoring-platform-team, 10Wikilabels, 10Easy, 10Google-Code-in-2017: Add accelerator keys (keyboard shortcuts) for forms in Wikilabels - https://phabricator.wikimedia.org/T162109#3786904 (10Aklapper) Imported as https://codein.withgoogle.com/tasks/5658007033610240/ [21:41:23] 10Scoring-platform-team, 10Wikilabels, 10Google-Code-in-2017: Provide a pytest for database of wikilabels - https://phabricator.wikimedia.org/T179014#3786939 (10Aklapper) [21:42:16] 10Scoring-platform-team, 10Wikilabels, 10Google-Code-in-2017: Provide a pytest for database of wikilabels - https://phabricator.wikimedia.org/T179014#3710279 (10Aklapper) Imported as https://codein.withgoogle.com/tasks/6463878189809664/