[14:00:05] o/ [17:28:09] o/ guillom & HaeB. Did we rope one of you into managing IRC during the showcase today? [17:29:42] halfak: roping was done at https://phabricator.wikimedia.org/T140427 ;) [17:30:43] Great! [17:30:44] :) [17:30:54] HaeB, do you have relevant super powers to change the topic? [17:35:37] halfak: no (i'm just voiced but not an op). but it seems you are op? ;) /msg chanserv access #wikimedia-research list [17:38:30] halfak: tried "/topic ..."? (leila and J-Mo seem to be ops too) [17:38:45] just started a meeting. Will fix this before the showcase starts. [17:39:43] HaeB, I think I just got it [17:48:07] halfak: yep; just joined the hangout to make sure everything was working fine [17:48:21] I should be in shortly. [17:48:29] BTW, guillom, you are now a god ;) [17:48:38] *op [17:48:44] :D [17:49:05] Oh. I was wondering what feats had suddenly warranted godhood! [17:49:34] I'm hoping you'll be interested in doing some mopping. :D [17:50:03] With great power comes great housework. [18:03:00] The showcase will begin momentarily, we're waiting for one of the presenters to arrive. [18:06:25] Except I'm an idiot and it won't start for 25 minutes :) [18:06:25] ^ Actually, we'll start in about 25 minutes [18:06:28] Stay tuned! [18:22:49] The showcase will begin in a few minutes (for realz this time). [18:33:26] what is the beatles image conspiracy theory? :) [18:33:44] paul is dead [18:34:24] oh [18:34:30] totally forgot about that one [18:48:16] This looks like a Glen Beck blackboard [18:50:35] I'm having trouble with the audio from the youtube stream, it's comprehensible but only with a good deal of concentration [18:51:13] yes [18:51:48] is wikidata used for the data? if not, why not? [18:55:03] SMalyshev: Do you want me to ask? [18:55:31] guillom: sure, would be interesting to hear [18:55:35] Sure [18:56:13] brendan_campbell, ^ see concerns about youtube audio quality above [18:56:20] if i understand it well - i just have to put in my scam a link to wikipedia? that will make it "correct"? [18:56:29] guillom: thanks [18:56:32] For some reason I dropped from the hangout :( re-joining. [19:02:15] seems we are still using the "decline" graph that ending in 2012 ;) [19:03:40] halfak: no graphs now? I can only see your face… [19:04:14] reloading the stream didn’t help [19:04:24] no slides here either [19:05:12] Aaron's slides : https://www.mediawiki.org/wiki/File:Deploying_and_maintaining_AI_in_a_socio-technical_system_--_Research_Showcase_(August_2016).pdf [19:05:36] guillom: would be great if you could poke someone and get Aaron’s screen share back [19:05:42] And the summer of research : https://meta.wikimedia.org/wiki/Research:Wikimedia_Summer_of_Research_2011 [19:05:45] Nettrom: ok [19:05:46] guillom: thanks [19:06:10] i didn't understand the ores documentation, and the software did not work in my browser [19:06:57] I'm reading the slides from meta [19:07:03] a bit silly but it's ok [19:07:21] :) [19:07:24] ok we're stuck in the time loop now [19:07:29] ah the endless echo loop [19:07:41] yes [19:07:43] this is going to be a wmf meme [19:07:45] ? [19:07:47] yep [19:07:48] I can see you talking yes, halfak [19:07:57] see you [19:07:58] i see your face now [19:08:15] yeah see slides now [19:08:15] yay, slides! [19:08:24] Ok, back to normal I think [19:08:32] now i see slides [19:08:32] now see slides [19:09:51] mishearing some fun things [19:10:11] what's the longest token they added -> what's the longest coconut added [19:10:24] thanks google hangout -> re-encoding -> youtube [19:11:59] i guess that time saved calculation assumes all patrollers trust ORES entirely (i.e. no longer check edits that ORES classifies as benign), correct? [19:15:15] * HaeB is spending a substantial part of his enwiki editing on watchlist and recent changes patrol, and has had ORES activated since last year - it's definitely useful, but i still look at non-flagged edits too ;) [19:17:42] q: i'm curious to know how much edit history was used to train ores [19:19:28] edsu: Will ask. [19:21:42] edsu: It seems that Aaron is going to address that shortly. Let me know if your question isn't answered. [19:22:39] guillom: will do [19:24:10] the problem of new and unregistered users... right [19:28:19] guillom: i had a question above too ;) [19:28:43] HaeB: Sorry, missed it. Will ask now [19:28:50] thanks! [19:29:47] another q, if there is time: but wasn't the model was built on human judgement (of edits) in the first place? how much is this bias against anons an artefact of the machine learning methods used, and how much is it judgment by the human scorers? (which we might decide to overrule by tweaking the algorithm) [19:35:17] Thank you all! [19:35:24] bye [19:35:27] \o/ [19:35:28] And sorry about that infinite echo. [19:35:30] Fun times [19:35:36] That was kind of funny [19:35:38] To be honest [19:35:39] one of the most amusing parts of the talk, that echo [19:35:40] echo [19:35:41] True. [19:35:42] ec [19:35:43] ho [19:35:55] But involuntary, I assure you :) [19:36:00] halfak: thanks for clarifying! so no actual revert data was used (as much related research has done)... wasn't certain about that point [19:36:04] I was amused that my prediction showed it self true by repeating itself. [19:36:15] :D [19:36:27] still, seems that there is a lot of useful information in those real-life reverts ;) [19:36:30] HaeB, yeah. That's right. We try our best to steer clear of past judgements. [19:36:38] E.g. was this user warned before [19:36:45] I'm pretty interested to see if ores classifications can be tested on other groups of editors. I guess you would have to go with registered users for these as they would have had to self-identifiy, but [19:36:48] Since that warning could have had bias and we don't want to create pile-ons. [19:37:00] edits based on declared gender, on declared ethnicity etc [19:37:27] to see if the training ste had bias or not [19:37:33] *training set [19:37:40] apergos, yeah. I like this idea. I've been reading about the Disparate Impact (US Law) recently, and I think that looking at these "protected classes" is a powerful strategy [19:38:01] it could at least let you catch those ahead of time [19:38:17] regarding the fairness point, there will be different conceptions here... many in the community think that editing anonymously is a deliberate and free choice. you decide to give the patrollers less information about your editing history, and in turn it's only fair that you get more scrutiny [19:38:23] and bear in mind that these classes are going to differ from one wp to another [19:38:44] HaeB, that's a good point that I didn't think of. [19:38:57] I mean on en wp the categories considered 'protected' or 'semi-protected' classes under us jurisprudence is probably good enough [19:39:16] where those categories will be the usual suspects (POC etc) [19:39:22] on es wiki? [19:39:37] on tr wiki? who da fuck knows [19:40:11] yeah. You'd need local collaborators for sure. [19:40:20] And it turns out we've been developing a lot of them :) [19:40:26] sweet [19:40:29] A positive side-effect of needing help with language assets [19:41:23] HaeB: the anonymous editor is probably not going to agree with that assessment [19:42:12] for a long time I edited anonymously. know why? I didn't have to agonize over language issues, I could edit without freaking out about L2 use [19:42:33] did this mean I consented to "heightened scrutiny"? hell no [19:42:46] and that's not even the usual case of the drive by editor [19:44:00] apergos: disagree with what? the statemenent that everybody can create an account? [19:44:30] "you decide to give the patrollers less information about your editing history, and in turn it's only fair that you get more scrutiny" [19:44:45] disagree with that notion [19:48:30] Yeah. I can definitely understand that. It's not like you show up with a notion of how skeptical anyone needs to be anyway. [19:48:32] apergos: well, it's not their volunteer work that would be made more difficult /their time that would be wasted, it's the patrollers' (assuming that they are not patrolling themselves, which a few IP editors do of course) [19:48:52] I understand that editors who do this curating would feel overburdened [19:48:57] s/would/do/ [19:49:11] I mean it's clear they have been feeling overburdened for a good long time already [19:49:36] it's even worse on the small projects [19:50:58] halfak: awesome presentation, thanks! [19:51:08] yeah it was pretty great [19:51:13] thanks edsu :) [19:51:28] & apergos [19:51:45] Still, we should be able to reduce the burden substantially without the bias. [19:52:02] If we can push prediction accuracy in the case of anons, that will only reduce the burden further. [19:52:04] i think it would be better to assess this from the overall perspective of our mission: how much does wikipedia and free knowledge benefit from allowing anonymous contribution (which is highly unusual on the internet, on most UCG sites you can't even contribute without providing an email address)? how much would it benefit from treating anons and logged-ins [19:52:04] strictly equally? [19:52:26] HaeB, 15-20% of productivity would go away [19:52:28] i'm of the strong opinions that anonymous contributions should be enabled [19:52:41] Oh wait... not sure about equal treatment, but allowing anons. [19:52:43] (and have argued for that several times in the past) [19:52:57] Not sure how equal treatment affects this. [19:53:01] but the treatment.. [19:53:37] halfak: not a youtube problem, but rather a microphone problem. have you all considered having this event in the lounge? i've suggested it before [19:53:55] we are talking about this as these are two different sets of people where an individual can't easily cross borders (like we do about gender, race, nationality) [19:54:09] if the event is scheduled in the lounge, it goes on my schedule, i do sound for the entire event, we use close microphones with a DSP that cancels echo [19:54:10] Understood, brendan_campbell. Which lounge? [19:54:12] better cameras [19:54:16] better presentation space [19:54:23] The big one on 5th? [19:54:28] ...but the same peopel who are editing as IP on WP have probably created accounts on a zillion other websites where it's required [19:54:33] halfak: Yep [19:54:50] anyway, GTG - great presentation as always, halfak [19:54:59] brendan_campbell, we did that a couple of times, but I think we switched primarily because the in-office attendance was too low to justify it. [19:55:07] But if you'd like us to switch anyway, I'm game. [19:55:13] Thanks HaeB. o/ [19:55:46] FWIW, my comment re. forcing registration was based on a couple controlled experiments done with the Growth team. [19:55:50] our conference rooms are not designed for broadcast-quality audio. It's just impossible to control in that space. I wouldn't worry about low attendance! [19:55:58] Tell anons to register and they leave and take their productive edits with them. [19:56:18] brendan_campbell, Cool. Will bring this up at our next research staff. Thanks [19:56:28] Appreciate it. Thanks for your patience [19:57:18] well most of the drive by editors would leave certainly [19:57:29] and folks who were committed to anon editing would also leave [19:57:36] so you don't get a lot of folks left [19:57:48] anyways [19:58:17] apergos, we see massive losses in productivity, so it's not just the drive-bys [19:58:30] My sense is that a lot of anons work that way on purpose. [19:58:40] yeah that's the 'folks committed to anon editing' [19:58:49] halfak: which experiments? [19:58:56] they either have specific reasons they want to be anon on wp [19:58:57] halfak: i wanted to ask what the feedback mechanism is like, in ores -- but i guess i can just try it out to see eh? [19:59:14] or they don't like to have accounts generally [20:00:02] edsu, no direct feedback right now. We actually don't want your ORES-tainted evaluations ;) [20:00:12] Oh! Wait [20:00:16] Qualitative feedback :) [20:00:23] Yeah... That depends on the wiki [20:00:39] A lot of communities want it on their local wiki. Otherwise, we have spaces set up on Meta. [20:01:20] so are the ores scores incorporated into an existing revision monitoring tool? [20:02:04] edsu, https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service#Tools_that_use_ORES [20:02:30] The best option right now is the ORES review tool if it's available on your wiki. [20:02:39] https://www.mediawiki.org/wiki/ORES_review_tool [20:03:11] on a lark i looked at this edit https://en.wikipedia.org/w/index.php?diff=734948210&oldid=734942388 [20:03:29] Secondarily, https://github.com/he7d3r/mw-gadget-ScoredRevisions, https://meta.wikimedia.org/wiki/User:Krinkle/Tools/Real-Time_Recent_Changes or http://tools.wmflabs.org/raun/ [20:04:20] edsu, yeah. that one gets 75% "damaging" -- which is probably because of the big content removal [20:04:20] ores said it was damaging with 74% likelihood? [20:04:29] It's not the strongest "damaging" prediction you'll get [20:04:43] Probably fits nicely into our acceptable false-positives [20:10:44] Now that I think of it, this is exactly the kind of edit our hashing vectorization work might be able to catch as good. [20:12:52] OK. I've got to go run an errand. I'll be back in about an hour. [20:13:00] Woops. Fixing topic first [20:16:04] I would say that edit was damaging [20:16:44] I mean, it wasn't bad-faith, but it was still damaging [20:23:57] Emufarmers: really? [20:24:09] edsu: look at the previous edit [20:24:26] vandalism is easy to revert [20:24:39] half-reverted vandalism tends to be a lot harder [20:24:45] the edit being analyzed is the revert isn't it? [20:24:50] no [20:25:20] the revert is https://en.wikipedia.org/w/index.php?title=Fertile_Crescent&diff=prev&oldid=734962564 :> [20:27:19] it doesn't look like either of those are actual reverts [20:27:43] i'm doubly confused because you are the editor on that last one :) [20:28:23] an editor vandalized the intro. The edit you linked was someone removing the vandalized material entirely. My edit was restoring it to its previous state [20:29:01] Emufarmers: ahh, i understand now [20:30:13] that's an interesting problem; had never considered half-reverted vandalism [20:30:32] thanks for pointing it out :) [20:36:47] unfortunately it's a tricky issue, because that sort of thing is really hard to detect and fix once it has sat around for a while (in this case someone would have had to see that the article was missing an intro and then think to look at the edit history). I think ORES is really promising for this [20:45:44] o/ yuvipanda [20:46:31] I'm trying to make some estimates. How much time per week would you say you spent on ORES when you were advising/contributing? [21:16:13] halfak, http://datworkshop.org/ from https://freedom-to-tinker.com/blog/randomwalker/the-workshop-on-data-and-algorithmic-transparency/ [21:16:19] hey halfak [21:16:31] I find it hard to do a 'per-week' since it was very spiky I guess [21:26:25] hi kjschiroo :) I THINK I'm installing the current version of WikiChatter and still getting the error. Can you check this? http://paws-public.wmflabs.org/paws-public/User:Jtmorgan/Teahouse/THQ-spot-checks.ipynb#Installing-newest-version-of-WikiChatter [21:27:30] Hi J-Mo, I'll take a look [21:27:58] thanks. not urgent, btw. [21:28:48] * yuvipanda spots a paws-public link in the wild :D [21:45:41] J-Mo, I do see that there is a problem now. I will look into what is causing it. [21:45:54] thank you, kjschiroo! [22:01:37] yuvipanda: there's more ;) https://phabricator.wikimedia.org/search/query/EMPlsOpCjrxn/#R [22:03:05] \o/ awesome :D [22:03:08] yuvipanda, fair point. Hmm. Will try to work with that. [22:04:04] subbu, looks interesting. I should probably popularize ORES with this community since they keep demanding open algorithmic practice to aid in their studies. [22:04:18] Would be nice to have a crowd of researchers studying bias and other problems in ORES. [22:06:12] yup, i figured you would find that interesting. [22:19:11] J-Mo, it looks like there is an issue with mwparserfromhell. I posted more details in your issue. [22:22:08] thanks kjschiroo!