[05:21:09] 10Scoring-platform-team (Current), 10ORES, 10Wikidata, 10User-Ladsgroup: new ORES labeling campaign for Wikidata - https://phabricator.wikimedia.org/T195701#4304360 (10Ladsgroup) To make the dataset (sorta) balanced, we automatically mark edits made by users with more than 1K edits as trusted and doesn't n... [11:34:04] (03CR) 10MarcoAurelio: [C: 031] "I guess that once merged we'll have to check for dupe namespaces on the Beta wikis, right?" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/440841 (owner: 10Awight) [11:52:35] (03CR) 10Awight: "MarcoAurelio @ 1:34 PM" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/440841 (owner: 10Awight) [12:37:47] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4305121 (10awight) How about this proposal: * Add optional `comment`, `guid`, and `origin` fields to judgments. `judgment.comment` is intended to hold a justification of t... [12:38:40] 10Scoring-platform-team (Current), 10ORES: Experiment with LIME integration for ORES, providing explanations for its predictions - https://phabricator.wikimedia.org/T196475#4305126 (10awight) [12:44:17] 10Scoring-platform-team (Current), 10ORES: Experiment with LIME integration for ORES, providing explanations for its predictions - https://phabricator.wikimedia.org/T196475#4305132 (10awight) [14:38:14] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Add wp10 model in PetScan - https://phabricator.wikimedia.org/T197871#4305369 (10Theklan) [14:39:25] 10Scoring-platform-team, 10articlequality-modeling, 10artificial-intelligence: Create a bot for automatic grading articles in euwiki using wp10 model - https://phabricator.wikimedia.org/T197872#4305381 (10Theklan) [15:00:04] o/ [15:00:08] hey [15:00:22] Back home, I hope? [15:01:41] halfak: Do you have any of the analysis code from https://meta.wikimedia.org/wiki/Research:Patroller_work_load kicking around? [15:01:57] Yup. Back home. [15:02:01] Oh goodness. No way. [15:02:02] Or, do you know of any more recent research on patroller workflows? [15:02:05] hehe [15:03:28] I'm thinking about user research, and realized that our initial goal shouldn't actually be to create a new workflow, but to transparently fit into existing ones. [15:03:31] Hmm... https://www-users.cs.umn.edu/~halfaker/publications/Accept_Decline_Postpone/schneider14accept.pdf [15:03:38] Is that the kind of thing you are looking for? [15:03:41] nice, ty [15:04:03] Not exactly, but it's interesting background reading. [15:04:28] https://meta.wikimedia.org/wiki/Research:Autoconfirmed_article_creation_trial [15:04:30] Maybe? [15:04:37] You might be looking for vandal fighting? [15:04:41] What I'd love to find, but doubt exists, is stuff like a count of active patrollers, their user-agents for patrolling actions [15:04:50] That's not usually called "patrolling" [15:04:55] hrm [15:05:00] RCP and NPP, I suppose [15:05:15] the people who will be using tools that integrate with JADE [15:05:33] Right. So, there's no "patrol" actions for RCP [15:05:35] gotcha. OK [15:06:27] I hear that RCP doesn't leave logging.log_action=patrol in enwiki, but they do in frwiki, for example. [15:07:58] thanks for pointing out patrolling vs. counter-vandalism [15:08:54] The overlap is weird, I see that each concept has its own top-level pages, but link to most of the same low-level tools? [15:13:24] https://www-users.cs.umn.edu/~halfaker/publications/When_the_Levee_Breaks/geiger13levee-preprint.pdf [15:13:40] http://stuartgeiger.com/papers/cscw-sustaining-order-wikipedia.pdf [15:13:59] enwiki specific but useful references. [15:14:52] Fantastic! I might as well take the chance to self-educate [15:15:19] :D paws is working today! [15:17:32] Do you think we can message the top 100 or so patrollers and vandal fighters, to ask them what tools they're using? [15:21:03] awight: it wasn't yesterday? [15:22:06] chicocvenancio: I haven't used it in a while, unfortunately I seem to always choose days that something is broken. Great to see it working so well now :-) [15:23:50] :) I've taken up maintenance for PAWS, if it does break, please ping me. In the last few months a lot of bugs were fixed and once we push the jupyterhub upgrade it should be a lot more stable [15:24:46] Very cool! It's really convenient for graphing database things, among other stuff. Thanks for doing this work! [15:27:23] awight, I think we could reach out to a few patrollers effectively, yeah :) [15:27:37] <3 chicocvenancio [15:27:57] PAWS is an amazing tool. I really want to see it under heavy dev again -- so we can close the gap between wikis and PAWS :) [15:28:04] Thank you for keeping it alive! [15:30:46] you're welcome. There are a few changes that I'll be pushing soon to it, I'm writing a post about it now. [15:36:34] Great to hear. :) [15:36:47] halfak: Maybe the web logs have a user-agent for the tools interacting by API, at least... [15:38:16] awight, maybe. that'll be a stretch. You can usually identify reverting tools by the traces they leave in edit comments. [15:38:28] See the methods in the "When the Levee breaks" paper [15:38:49] I likely have a pile of regexes somewhere ... [15:52:20] halfak: I just sent you an email about a very interesting, potentially relevant, talk I saw yesterday [15:52:41] * halfak looks [15:58:37] Looks interesting. The abstract doesn't say any take-aways. >:( [15:59:09] * halfak curses some qualitative researchers' tenancies to hide their conclusions [15:59:28] Nope, sorry! I think they're going to share their slides with us, I'll try to get permission to share them (although, I think it might be interesting to invite them to talk at a research meeting, if it's something you think would be a good fit) [15:59:35] ewhit_, do you think we could use or build on that work? [15:59:36] looool forever at that, it's v true [16:00:09] yes, absolutely. Not necessarily on my specific project rn, but the foundation in general, and with any work going forward [16:00:11] gtg for a while o/ [16:00:17] o? [16:00:20] o/ [16:00:36] I thought Katta's work was great, and very relevant [16:01:33] We might be able to build on some findings for my current project, but I would hesitate to do that before the work is *officially* published [16:06:51] ewhit_, if it's accepted to CSCW, there's no good reason to not talk about it. Was it a round 1 paper? (that was concluded in January, I think) [16:12:25] No, it's still under review [16:22:30] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10artificial-intelligence: Enable ORES filters on srwiki - https://phabricator.wikimedia.org/T195870#4305757 (10Acamicamacaraca) 05Open>03Resolved Duplicate [16:49:58] 10Scoring-platform-team, 10Wikilabels: Labeling campaign for dewiki shows 'revision not found' for all revisions - https://phabricator.wikimedia.org/T194457#4305813 (10Cirdan) No, it still doesn't work for me... I just went through a batch of 50 revisions and not a single one could be retrieved... For example,... [18:11:47] AFK for lunch [18:19:22] (03CR) 10Umherirrender: "namespaces are case insensitive, but you have to look for the talk space" [extensions/JADE] - 10https://gerrit.wikimedia.org/r/440841 (owner: 10Awight) [19:52:54] back! [19:52:57] forgot to say [19:57:34] same here. We should commit to another round of keeping /away up-to-date, it's been really helpful for me. [20:11:03] Weird, the histogram seems to be steeper than a power law [20:11:14] a log plot still looks exponential. [20:12:50] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4306270 (10Halfak) If we keep endorsements, I think the origin might make more sense in the endorsement since you could get endorsements from different origins and that wou... [20:22:21] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10artificial-intelligence: Enable ORES filters on srwiki - https://phabricator.wikimedia.org/T195870#4306294 (10Halfak) @Acamicamacaraca, can you link to the duplicate task? [20:23:13] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4306296 (10awight) Okay, let's drop endorsements and move to talk. I don't quite catch what you're saying about "origin", I am proposing that the judgment is accompanied b... [20:26:51] https://paws.wmflabs.org/paws/user/Adamw/notebooks/Patrolling%20analysis.ipynb?redirects=1 [20:28:33] awight: http://paws-public.wmflabs.org/paws-public/User:Adamw/Patrolling%20analysis.ipynb ? [20:29:09] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), 10artificial-intelligence: Enable ORES filters on srwiki - https://phabricator.wikimedia.org/T195870#4306324 (10Halfak) 05Resolved>03Open I searched hard and couldn't find something. I'm... [20:29:21] chicocvenancio: o/ I was lobbing at halfak, mostly [20:29:41] 10Scoring-platform-team (Current), 10ORES, 10Wikidata, 10User-Ladsgroup: new ORES labeling campaign for Wikidata - https://phabricator.wikimedia.org/T195701#4306326 (10Halfak) We don't actually count all edits by people with 1000+ edits as good. We'll check to see if the edit was reverted and if they are,... [20:30:26] I figured, but only you can see through that link [20:30:33] ooh [20:30:46] 10Scoring-platform-team (Current), 10ORES, 10Wikidata, 10User-Ladsgroup: new ORES labeling campaign for Wikidata - https://phabricator.wikimedia.org/T195701#4306340 (10Halfak) How big of a sample do you think we would need in order to get enough "needs_review" samples? [20:30:48] Cool, thank you! [20:30:55] chicocvenancio i can see it :) [20:31:31] https://paws.wmflabs.org/paws/user/Adamw/notebooks/Patrolling%20analysis.ipynb this? [20:31:45] yep [20:31:49] oh [20:31:51] not that one [20:31:55] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4306348 (10Halfak) Arg. This just looks like endorsements in a messy structure to me. :| [20:31:58] but http://paws-public.wmflabs.org/paws-public/User:Adamw/Patrolling%20analysis.ipynb [20:32:15] yes! that was my point [20:32:30] * chicocvenancio got extremely worried something was very wrong for a second [20:32:56] heh [20:33:06] chicocvenancio though https://paws.wmflabs.org/paws/user/Adamw/notebooks/Patrolling%20analysis.ipynb shows a white screen [20:33:22] it should probaley throw a 404 or a http error that indicates permission denied [20:46:53] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4306416 (10awight) If we're dropping endorsements for the first iteration, let's also not try to simulate it using the multiple judgment pattern in my last example, then, a... [20:49:31] The patroller histogram just got interesting. A histogram of the top thousand most productive patrollers over the past year shows a power law until the last 200 or so, when the power coefficient increases. Probably a normal thing to see with these? [20:53:53] 10Scoring-platform-team (Current), 10Documentation, 10Easy: Document JADE judgment structure - https://phabricator.wikimedia.org/T179301#4306423 (10Halfak) Re. origin. I'm imagining the following scenario: 1. A huggle user marks an edit as damaging and bad-faith 2. A judgement is created for the edit and t... [20:57:29] 10Scoring-platform-team, 10revscoring, 10Chinese-Sites, 10artificial-intelligence: Chinese language utilities - https://phabricator.wikimedia.org/T109366#4306427 (10Halfak) back to it1 I'm looking through the bad and informal words. I should have a PR soon. [20:58:25] Lool I just got to the section of When the Levee Break that talks about "trace ethnography"--it's like going through people's garbage cans... [20:59:41] http://stuartgeiger.com/trace-ethnography-hicss-geiger-ribes.pdf [21:02:07] aargh I think this won't work. halfak: you were analyzing revert edit comments, right? [21:02:28] Right [21:03:04] We only looked at where people took an action. Lots of patrolling is seeing that something is good and moving forward. [21:03:58] exactly, I think I want the whole picture and limiting to just reverts will give me a big skew in terms of the tools used. [21:04:30] Not that I'm being picky about data quality... I just want to see what the most common workflows are. [21:04:34] halfak: while I'm thinking about Q1 goals for the cloudy half of my job, want to set up a time to talk about my Q1 goals for scoring platform? [21:04:43] :D [21:05:08] harej, good Q! Yes. [21:05:21] To help me out, what's an example of a good goal from your perspective? [21:05:31] I'm also happy to come up with proposed Q1 goals, but for that I need to know what direction my thinking should head in. [21:05:38] ...why did I take off my watch [21:05:41] You could give me a goal you have set in the past. [21:06:36] It seems that addressing some of the JADE questions that awight and I have been working through could be a good goal. [21:06:58] Or maybe getting the goddamn thing deployed for real this time? [21:07:25] I'm itching to see it in use [21:07:47] as you can see by my scratching around in the patrol logs garbage can [21:09:13] awight, maybe we can set the scalability problem asside without all of this digging. [21:09:15] What do you think? [21:09:22] Is this necessary in order to move forward? [21:11:20] not at all necessary for scalability, I'm doing this mostly for fun but also because I think it'll be helpful to have this as background guiding user research, and designing our features to be compatible with the common tools [21:13:45] Also, having an idea of the "share" of patrolling+CV for each tool might be useful for integration and adoption planning. [21:14:25] halfak: well, for example, my Q4 goals were to finalize the toolhub data model, come up with wireframes, and finalize the initial feature set for toolhub [21:15:04] awight, feature set for JADE's MW integration? [21:15:05] i guess before we talk about my Q1 goals we should set up time to talk about the current scoring platform product offerings, their levels of maturity, and what needs to be done [21:15:11] That sounds like a solid chunk of work [21:15:25] Will involved working with J-Mo as he's set time aside for some user-research [21:16:02] right. I think it makes the most sense to have you look at JADE and WikiLabels at first. Looking at product management of ORES is a longer conversation, I think. [21:16:16] hargh, I can't find an example of hive queries against wmf_raw.ApiAction [21:17:02] harej, are you working from the office? [21:17:09] Not today (or tomorrow) [21:17:22] ok cool. I just put something on your calendar for tomorrow. [21:17:27] And did not reserve a room [21:17:28] :) [21:17:42] Also, harej, I borrowed your desk while I was in the office :D [21:17:49] Good! I want people to. [21:18:01] I hope you enjoyed the accoutrements of my desk, including the picture of my sister's dog. [21:18:05] Was working with leila so it was perfect [21:18:16] "This is not a whiteboard recycle bin" [21:18:18] lol [22:00:26] nice. so I have a decent sample of API requests with action=patrol... [22:00:48] A lot are from browsers, I assume a gadget. [22:04:16] Seems that Huggle doesn't hit that endpoint, so it's just used to revert and rollback? [22:06:50] ./src/huggle_core/apiquery.cpp: this->actionPart = "patrol"; [22:07:06] * awight shrugs. de heck [22:07:27] o/ [23:46:06] halfak: have you thought about ML as a service? [23:46:12] like, a service for training new models?