[03:55:01] 10Huggle, 10WorkType-Maintenance, 10good first bug, 10patch-welcome: look for new messages on all projects - https://phabricator.wikimedia.org/T70502 (10Aklapper) @Craftword: Hi! If you still work (or plan to work) on this task, [is there anything that we could help with](https://www.mediawiki.org/wiki/How... [18:41:03] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by ToBeFree (mobile) link https://en.wikipedia.org/w/index.php?diff=927318652 edit summary: /* Warning user */ not specific enough [19:04:21] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by Interstellarity link https://en.wikipedia.org/w/index.php?diff=927321552 edit summary: /* Warning user */ providing an example [19:05:22] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by Interstellarity link https://en.wikipedia.org/w/index.php?diff=927321684 edit summary: fix link [19:05:59] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by Interstellarity link https://en.wikipedia.org/w/index.php?diff=927321748 edit summary: /* Warning user */ Spelling/grammar/punctuation correction [19:07:19] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by Interstellarity link https://en.wikipedia.org/w/index.php?diff=927321920 edit summary: /* Warning user */ note [19:07:37] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by Interstellarity link https://en.wikipedia.org/w/index.php?diff=927321958 edit summary: /* Warning user */ Spelling/grammar/punctuation correction [20:10:50] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by ToBeFree link https://en.wikipedia.org/w/index.php?diff=927329722 edit summary: /* Warning user */ replying [20:24:33] o/ petan [20:24:59] I dropped you an email about getting that dataset of judgments from Huggle. Is there somewhere I could download it or copy it from toolforge or something? [20:31:27] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by Interstellarity link https://en.wikipedia.org/w/index.php?diff=927332384 edit summary: /* Warning user */ Replying to ToBeFree (using [[w:en:User:Enterprisey/reply-link|reply-link]]) [20:33:49] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Feedback was modified, changed by ToBeFree link https://en.wikipedia.org/w/index.php?diff=927332666 edit summary: /* Warning user */ yay :) you're welcome [21:23:58] halfak: hello gonna check it now... [21:24:14] frankly speaking I didn't do much Huggle stuff lately [21:25:46] halfak: it's a good question, we used to have a postgres db in wikimedia cloud but that was years ago I doubt it's alive anymore [21:26:08] another source would be the IRC channel where most of Huggle clients report this to, but that's not gonna get you historical data [21:26:14] let me check it [21:26:20] maybe the DB is still existing [21:26:33] Ooh. I hope so. We could use that data. [21:26:55] We're finally getting close to launching a central repository of this stuff. [21:33:38] wow it still exists :D but I am not sure if this is really going to be that much useful, because this DB only contains a list of edits flagged as "suspicious"... the list of reverted edits can be gathered from the wmf DB though [21:34:19] I will PM you access to DB [21:35:17] petan, do you have any data on people deciding that an edit is good/vandalism/goodfaith damage? [21:37:59] I'd nearly forgot about "suspicious". I'd like to capture that too. [21:38:06] I think this is something we can gather from the IRC logs only right now, but it shouldn't be big deal to implement some other form of gathering this kind of data [21:38:17] although I really would like to avoid "spying" on users [21:38:37] Do you have a log of the channel? [21:38:56] That's a fair point. Though I imagine that this all turns into public data though, right? [21:39:11] E.g. the warning message is different for goodfaith damage and vandalism. [21:39:20] I guess marking something as "good" isn't really public. [21:39:36] no it's not, I don't really consider that logging into IRC channel as "public data" [21:39:57] here is a privacy statement BTW, which describes what all data Huggle collects and how: https://en.wikipedia.org/wiki/Wikipedia:Huggle/Privacy [21:39:57] Yeah. I was thinking that it turns into edits on the wiki. [21:40:19] I suppose it doesn't matter that I don't intend to re-publish this. [21:40:25] the channel is not being logged right now, it could be though [21:41:32] My primary goal is to explore strategies for turning edits that ORES mistakenly predicts to be damaging (or mistakenly predicts are not damaging) into bug reports that we can work with. [21:41:43] I wonder if we could drop user information from the judgment. [21:41:55] I don't think I need that. [21:43:55] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Privacy was modified, changed by Petrb link https://en.wikipedia.org/w/index.php?diff=927340947 edit summary: reworded [21:44:54] Change on 12en.wikipedia.org a page Wikipedia:Huggle/Privacy was modified, changed by Petrb link https://en.wikipedia.org/w/index.php?diff=927341061 edit summary: typo [21:45:54] halfak: yes I don't have a problem with that, although if we start doing that, I would definitely mention that in that privacy statement... I am just wondering what's the best way to do that [21:46:29] one would be to gather the data from IRC channel, that is already possible and working with all versions of Huggle, another would be expanding the current ORES extension [21:46:38] Yeah, it seems like having a public discussion about it would be helpful. I'd be interested in helping with that. [21:46:45] https://github.com/huggle/extension-scoring [21:47:22] I want to work towards getting judgment data from all patrolling tools into the same repo. [21:47:44] I mean - what's the best way to gather the data you need, not what's the best way to modify that privacy statement, that's probably not so big deal, most of the work done on wiki is kind of publicly track-able anyway [21:48:02] Right. [21:48:04] Gotcha. [21:48:14] BTW, what channel are huggle activities logged in? [21:48:35] ok do you have any interface / API to collect these metrics or to store them in that "repository" [21:48:48] It'll be a mediawiki API endpoint. [21:49:07] We'll have an endpoint that you can dump information into. Let me get the spec... [21:49:42] it's on my own IRC network due to very strict limits imposed by freenode, it's mentioned on that privacy statement page, server address is irc://irc.tm-irc.org/#en.wikipedia.huggle [21:49:55] for english wikipedia [21:50:13] The spec of the endpoint is here: https://phabricator.wikimedia.org/P8830 [21:50:16] here is documentation for it: https://en.wikipedia.org/wiki/Wikipedia:Huggle/HAN [21:50:21] See the first section for "proposeorendorse" [21:51:03] this is how the actual data in IRC channel look like: [21:51:04] (Thu Nov 21 22:13:56 2019) Incoming CTCP for #en.wikipedia.huggle from CAPTAIN_RAJU!huggle@79BF4485.EECC3F62.D583DFB1.IP: ROLLBACK 927337505 [21:51:05] (Thu Nov 21 22:13:57 2019) Incoming CTCP for #en.wikipedia.huggle from CAPTAIN_RAJU!huggle@79BF4485.EECC3F62.D583DFB1.IP: WARN 1 64.88.209.226 [21:51:33] that means they did a rollback of revid 927337505 and sent a level 1 warning to 64.88.209.226 [21:52:06] but to be honest these data can be extracted from WMF DB as well [21:52:20] Huggle is flagging its own edits so it's easy to select them [21:52:38] what you are probably interested in are "user decisions" which are not written into the wiki [21:52:40] Can you tell from the IRC message if they thought it was vandalism or good-faith? [21:52:48] Right [21:53:54] which are for example suspicious edits, that are both in HAN and that DB I talked about... regarding the good-faith / vandalism, well that's something that could also be selected from WMF DB as "good faith" reverts have different edit summary, but I don't know how much people actually use that feature [21:54:06] I believe most of users just hit revert button even if it's "good faith" [21:54:48] lol damn it. [21:54:52] ha. [21:55:05] Are there any log lines for when someone decides an edit is good? [21:58:03] I'm kind of surprised to not see any log activity in #en.wikipedia.huggle yet. [21:58:06] Slow day? [21:58:53] Oh! petan, by the way I was digging through the history of Huggle reverts on (I'm pretty sure) French Wikipedia recent and I didn't see huggle linked in the edit comments. [22:00:13] yeah, last activity was 10 minutes ago... not much going on right now I guess [22:00:58] "summary" is blank at https://fr.wikipedia.org/wiki/Wikip%C3%A9dia:Huggle/Config but it has [[WP:HG|HG]] in the enwiki equivalent. [22:01:01] regarding frwiki, the edit summary is customizable per wiki it's possible they decided to remove the link in the edit summary? [22:01:25] that page isn't being used anymore, latest Huggle uses .yaml [22:01:34] Aha! [22:01:36] oh [22:01:45] well they don't seem to have it on frwiki [22:01:51] there is a fallback to old format if it doesn't exist [22:02:03] this is actual enwiki config though https://en.wikipedia.org/wiki/Wikipedia:Huggle/Config.yaml [22:02:27] Gotcha. [22:02:29] That makes sense. [22:03:06] Say, one of the most important thing to me is knowing when someone looked at something and thought it was OK. It sounds like that's not something that gets recorded anywhere. Is that right? [22:03:56] well I am sorry that we don't seem to have any historical data for you that you could actually use, I am trying to remember that conversion we had about this, but it had to be a while ago, I am not sure if I promised you any existing data, or if I just said that it's possible to collect them... [22:04:17] what I definitely remember is that at some point I suggested Huggle could give a feedback back to ORES, there is even placeholder code for it [22:04:36] Well, that's definitely something we want to accept! [22:04:44] And we're working on it. [22:04:52] Anyway, Petan, you have been a huge help. Thank you! [22:05:16] I'll be getting back in touch with you some time in the next couple of month about testing out this repo. [22:05:37] Would you mind if I reached out in the shorter term to discuss some of our design decisions? [22:06:50] well I really didn't help you much... but yes, we could probably implement this feedback back to ORES or to your repo [22:07:16] Just telling me what is and isn't available is a huge help. [22:07:18] regarding "someone thought edit is OK", that would be "good edit", it's missing in docs, but it also get reported to IRC channel [22:07:19] :) [22:07:28] Oh awesome! [22:08:12] Change on 12en.wikipedia.org a page Wikipedia:Huggle/HAN was modified, changed by Petrb link https://en.wikipedia.org/w/index.php?diff=927343452 edit summary: /* How it works */ GOOD [22:08:26] it was so long when I last worked on Huggle I wasn't even sure we report it [22:08:30] I had to dig it in code, we do https://github.com/huggle/huggle3-qt-lx/blob/master/src/huggle_ui/vandalnw.cpp#L199 [22:09:15] but I don't know how much people actually hit the "good edit" button, I guess it depends on their level of experience, I use it a lot, some people too, but some people don't even seem to be aware there is such a button [22:09:22] because in IRC logs I can see they never use it [22:09:44] petan, one more quick question before I run away. I am seeing a more minimal log in the #en.wikipedia.huggle channel. e.g., "WARN 1 Student696969699" Is that expected? [22:09:54] Oh! So "next" is different [22:10:15] I'd like to interpret that as "looks good to me" [22:10:57] Yes that's correct, what I sent to you was a whole line from scrollback from my IRC client, the timestamp and nickname was form my IRC client [22:11:15] the protocol is really simple just as described on that wiki page [22:11:23] https://en.wikipedia.org/wiki/Wikipedia:Huggle/HAN#How_it_works [22:11:34] it's just a command with parameters, like [22:11:39] ROLLBACK 12345 [22:11:55] Gotcha. That makes sense. [22:13:51] for the scoring extension in Huggle - as I said there are placeholder hooks for these events such as good edit, suspicious edit etc. I think it should report this back to ORES somehow, I think few years ago when we discussed this first there might not have been an API in ORES for data input yet [22:14:13] not sure if it's possible now [22:14:18] Right. It's still not quite ready yet. [22:14:21] I can add some web callback for these events [22:14:40] so that when people click "good edit" in Huggle it would report it somewhere, either ORES or that repository you talk about [22:14:52] I can report even reverts, suspicious edits, skipped edits etc [22:15:19] Judgments are complicated. We need people to be able to refute them and figure it out together. We need to deal with vandals submitting bogus judgments. So we're integrating this with MediaWiki so people can revert bad judgments. [22:15:23] skipped edits are not really logged anywhere right now - that's when user press spacebar, it literally is "I have no idea what's this edit, I don't want to deal with it, give me next edit" [22:15:57] lot of people use Huggle in a way they skip most edits and look for obvious vandalism [22:16:18] petan, that's interesting. I really thought about that as "this is probably good". If someone presses space on an edit, does the edit get shown to others? and will it stay in the queue forever if no one reviews it? [22:16:31] so I am not sure how much a list of edits that were skipped would help [22:16:41] That's a great point. [22:17:00] I wonder if that judgment of "This is not *obviously* bad" is useful though. [22:17:33] when someone hits spacebar, Huggle moves to next edit and reports nothing nowhere, which means the same edit most likely remain in other user's queues, only time when edit is remove from other people's queue is when it's reverted or flagged as good [22:17:46] when it's flagged as suspicious, its score is increased and it's moved higher in the queue [22:17:57] Ooh. That's interesting too. [22:18:15] See, I'd like to capture that suspicion and flag it in people's watchlists and in other tools too. [22:19:46] Even edits that are hard to judge. Suspicious or not, I want someone with some related knowledge/context to look at it. [22:19:50] that was the whole point of "HAN" the idea was to use some widely standard network protocol (such as IRC) to create some universal network that could be used to share this kind of data between similar tools to Huggle, but it was never really adopted by anything else, I created a connector for ClueBot though [22:20:28] Hmm. That sounds familiar. [22:20:53] btw the reason why suspicious edits are stored in that DB is that in past I had plans to create some web interface where people (experts on topic) could review edits that Huggle users marked as suspicious [22:21:12] I might want to have one of my grad students interview you about that. It's a fascinating idea. [22:21:59] But really, the next practical thing I can do is convince WMF product teams to build these kind of backlogs and integrations. [22:22:04] because many of these "suspicous" edits are edits that look like potential vandalism, but it's hard to figure it out unless you aren't an expert in the given topic. Typically, modification of birth dates, historical events etc [22:22:39] oh yes interview is not a problem [22:24:53] btw regarding bogus data is that really a big deal? [22:25:13] Good question. We've been burnt by it in the past. [22:25:16] when 1 out of 10 000 judgements is a bogus, does it really affect the prediction engine in any way? [22:25:33] Na. Our data is already way worse that that ^_^ [22:26:02] But re. getting burnt, I'd like to make this repo more public. [22:26:12] Essentially, I want people to use it to refute ORES directly. [22:26:29] So people will see when ORES thinks their edits are bad. That'd be a vector for vandalism. [22:26:35] Not quite sure how to surface that yet. [22:26:49] hmm I don't really know much about these AI engines, although it's really cool, it's a shame I didn't have enough time this year's hackathon for ORES [22:27:29] Hopefully we'll be working together at the next hackathon on integration ^_^ [22:27:48] "people will see when ORES thinks their edits are bad", you mean there would be some feedback in the edit form directly? [22:28:12] I don't think we'll have it in the edit form. But definitely on Special:Diff page. [22:28:24] We already have ORES scores showing up on Special:RecentChanges and Special:Watchlist [22:29:04] In the next couple months we're going to work on strategies for letting judgments show up there too. It could be a great mechanism for transferring "suspicious" [22:30:13] that's cool, can ORES calculate quality just for edit, or even for whole page? say for new articles? [22:30:30] you know there is this manual review process for new articles now [22:30:37] Yup! We've got a few predictions for new articles. [22:30:50] Yeah. I've been targeting that. It's not nearly as well supported as edit review. [22:31:20] We predict "Is this article Spam or Vandalism?", "What quality level is this article?", and "What is the general topic space of this article?" [22:31:49] With topic, we're thinking the same things you are re. "suspicious". You really want someone who knows something about the subject to check if something is a hoax -- or if it's notable. [22:34:53] very nice, I can say ORES is making much bigger progress than Huggle ;) but if you make a phab ticket for that feedback stuff, I might find time to implement that [22:35:15] as I said, the placeholders are in there, I just need to add some URL to send the feedback to [22:35:16] Sure! I'll make one right now, though there will be a lot of blockers. [22:35:51] Also, we're making different progress. In the world I'm working to build, Huggle and systems like it are very important. I want to build infrastructure that make them all work together and have common resources to draw from :D [22:36:35] sounds great [22:45:36] https://phabricator.wikimedia.org/T238877 [22:45:39] petan, ^ [22:46:18] I'll keep adding notes there as we make progress. I'm going to be thinking about this notion of "suspicious" and will try to get a proposal together for you to consider. [22:47:07] ok