[14:32:39] https://dl.acm.org/citation.cfm?id=3051116 [14:32:52] "Exploring the Impact of Trolls on Activity Dynamics in Real-World Collaboration Networks" [14:32:56] Looks interesting [14:33:04] Full text: http://papers.www2017.com.au.s3-website-ap-southeast-2.amazonaws.com/companion/p1573.pdf [14:38:50] * halfak clicks [14:39:27] (I haven't read it yet.) [14:39:37] Also, hello. [14:40:06] hey guillom :) Good timezone-apt-greeting [17:57:05] hello, research folk! I've got a little research project some of you might be interested in: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28proposals%29#Experiment_to_see_if_training_modules_are_helpful_for_new_users [19:30:05] neilpquinn, ^ [19:30:12] Did you see ragesoss's message. [19:30:23] AbbeyRipstra, ^ [19:31:05] I see that J-Mo has seen it and signaled support. [19:31:09] :) [19:31:35] J-Mo had penciled in plans to collaborate on this, but it got bumped off his work schedule. [19:31:53] ragesoss, wish I could help out too [19:32:00] Maybe I can consult come measurement time. [19:32:12] Just so long as you have a fair control :) [19:32:14] I'll probably pester Mako to help with analysis. [19:32:31] cool :) [19:33:07] halfak: control and experiment will just differ by whatever ruby randomly sorts users into, always in equal numbers from the same query of recent users. [19:33:33] https://github.com/ragesoss/ruby-ragesossbot/blob/master/process_new_users.rb#L31 [19:33:35] related, halfak: I've started experimenting with using ORES to identify potential Teahouse invitees. I'm thinking a good way to start would be to pass all the user's edits into https://ores.wikimedia.org/v2/#!/scoring/get_v2_scores_context_model using the goodfaith model. Should I then calculate their 'average' good-faith score from the individual scores? is there a canonical way to do this? [19:33:57] ragesoss, can I suggest you split users by odd and even user_id? [19:34:04] that makes querying later *way* easier. [19:34:46] J-Mo, there's a way to do this, but getting an average score will be an OK approximation. [19:34:54] figured. thanks Halfak [19:35:09] Let's talk about fitting beta distributions later :) [19:35:22] I'll give you a cute little python function to run on the scores. [19:35:33] halfak: really? why does that make such a difference if we're working from a discrete set of user ids anyway? [19:35:34] also, ragesoss: if you just want to calculate the impact on long-term retention, the method halfak developed for our Teahouse retention study is actually really easy to implement & analyze: https://meta.wikimedia.org/wiki/Research:Teahouse_long_term_new_editor_retention#Methods [19:35:40] I could probably help with that ;) [19:36:14] \o/ [19:36:46] ragesoss, it's really just about having to join the dataset to the user table later. If you don't need to do that, you can gather stats right from quarry without having to upload a table. [19:36:58] I'd rather poke my eyes out than try to get a new dataset available to quarry. [19:37:18] and if I do end up writing a ORES-based sampling script for HostBot, you can borrow that to identify people to invite. [19:42:06] halfak: how does using odd/even save from that, if it's not every odd/even user (or every one in a range)? And with ~5000 ids, can we just paste in the list of them to quarry? [19:42:59] Why wouldn't it be every odd/even user? [19:43:01] I've never done major analysis tasks like that on quarry, so I don't think I understand the typical hurdles. [19:43:16] ragesoss, you don't want to paste more than 100 IDs [19:44:05] because I'll only get permission to run the bot a certain amount, and it will only message people (or put them in control) who meet whatever criteria at the time of the current bot run. [19:44:43] like, between 1 and 5 edits is the current query I do to find new users. [19:46:51] ragesoss I would use PAWS, not Quarry. [19:47:29] actually, I know I said I didn't have time when we talked about this, like, 4 months ago. But I may have more time now :) want to hack together? [19:47:39] sure [19:48:37] :) PAWS is also a good option. I like to use both. I like to generate datasets in quarry (so that ger permalinks) and then load them into PAWS for analysis. [19:49:17] I just cleaned up HostBot (all it does is invite people to the Teahouse, now), and it would be pretty easy to incorporate your A/B test with the new talkpage invite. [19:49:30] do you Python? https://github.com/jtmorgan/hostbot [19:50:09] yeah, I do enough Python to hack my way through stuff. [19:50:19] my earlier bots were in pywikibot [19:50:28] that should be enough [19:50:38] but I decided that if I'm doing this myself, it's going to be in ruby [19:50:54] but the actualy bot logic is simple enough. [19:51:41] ah, well if you want to see how I do the same thing in Python, take a gander at the code & ping me if you have questions. [19:52:26] are you going to log invites in a MySQL db, or somewhere else? [19:52:43] yeah, I was going to do it in a SQLite db [19:53:17] HB has its own db on labs where it stores users it invited and control users? [19:53:26] it does! https://quarry.wmflabs.org/query/10027 [19:53:39] ah, neat [19:55:50] lmk if you want to collaborate on the implementation. I don't know Ruby, but can still help with study design if you want me to. [19:56:18] if you're up for helping, let's just do it with HB instead. [19:57:13] I have the messaging all set, and I'm not picky about the rest of it. [19:57:50] I can probably make myself free whenever you have some time; maybe let's pair on it? [19:58:06] on-wiki discussion is going well so far, so hopefully BAG won't be a blocker. [19:59:19] I stole some of the styling from the welcome templates HB has been using: https://en.wikipedia.org/wiki/Template:Welcome_training_modules [19:59:42] drop shadows FTW [20:02:12] nice [20:03:19] yes, ragesoss if all that's required is setting up a workflow for generating a daily sample, sending half of them a talkpage invite, and recording the results, this should not be difficult to do with HostBot [20:03:45] I'll create a new table in the hostbot db to log invites [20:03:58] ideally, I'd like to do it as close to their first editing session as possible [20:04:43] I was planning on doing many cycles per day, inviting (or control grouping) only the newest few users each time. [20:04:51] as long as we can still run it as a cron job (every hour, maybe?), that doesn't change the implementation [20:04:57] yeah, exactly. [20:05:03] a cron job was my plan. [20:05:32] I mean, once per day would work too, if we're just doing a few at a time. it'd just take longer to get up to sample size. [20:06:35] cool. then if you want to use HB you can just update the BAG request. Here is the list of previous approvals: https://en.wikipedia.org/wiki/User:HostBot#BAG_Approvals [20:09:12] I'll need to open a separate one for HB, I think, just cause all the templates and page names rely on bot name. [20:09:37] but we should add the code for this experiment first? [20:12:40] agree. [20:14:25] so ragesoss you want people who made their first edit since the last time the bot was run? do you care whether they are good faith or not? do you care whether they have edited the article namespace yet? [20:14:56] right. and don't care about either of those things. [20:23:36] cool. let's see if I can get this done in the next couple of days, ragesoss [20:23:48] awesomesauce [20:25:48] ragesoss didn't we create a placeholder page for this work on Research:Projects? [20:27:00] J-Mo, ragesoss: I just commented on the Village Pump without realizing you were still discussing here :) [20:27:11] My one suggestion is using a standard welcome template as a control instead of nothing. [20:27:33] neilpquinn: but we don't be default use any welcome templates for most users. [20:27:52] the point is to find something that is better than literally nothing. [20:28:09] ragesoss: True, but I think there are de-facto standards that a lot of welcomers use. [20:28:12] right, because enwiki put the cabosh on "welcome bots" long ago [20:28:31] because… who knows [20:28:43] it's true, but those rely on a human doing the welcoming, in most cases. [20:28:50] most people won't receive a welcome template within an hour of thier first session [20:29:08] and the reason why we don't use welcome bots is because no one has ever shown that a welcome bot is better than *not* being welcomed by a bot. [20:29:31] well, that's silly and wrong [20:29:36] ragesoss: has that research actually been done? If so my suggestion makes less senes. [20:29:43] but hey, you know, it takes all kinds [20:30:03] you can't do the research, because the decision was already made [20:30:23] neilpquinn: that's the received wisdom, anyway. it's the relevant null hypothesis to make the case for doing further work in this area, and finding an actually-really-useful way to automatically welcome newbies. [20:30:31] we only got HostBot through because it's an *invite bot* [20:31:12] ragesoss: well, it would be really nice to test that received wisdom, no? :) [20:31:18] and in this case it would be a "tutorial bot", not a "welcome bot", I guess. [20:31:25] although, yeah, three groups would be good. [20:31:28] of course it all depends on how much work or additional approval barriers it would add. [20:31:34] one normal welcome bot, one no template, and one this one. [20:31:59] that kind of muddies the study design [20:32:06] ragesoss: exactly, I think that would be even more interesting than your tutorial vs. nothing. [20:32:37] J-Mo: the analysis can still start exactly the same way. [20:33:05] sure, but do you want to find out if your tutorial works, or do you want to find out something else? [20:33:09] I mean, my intention was to do the simplest possible thing. [20:34:16] find out if the tutorial works. but, per above, my more general interest is on getting to the point where we have a useful onboarding experience for new users, in general. [20:34:45] I guess we could give it a go. I do think that bringing welcome templates into the mix may resurrect the whole "we don't do welcome bots" conversation and distract from the goal of the study [20:34:55] yeah. [20:35:13] J-Mo: I see the value of simplicity, but from my perspective it seems really really useful to just confirm our assumption that dropping a (sprawling) welcome template does nothing for retention. That would make it clearer that what matters is not mechanical, perfunctory outreach but actual, well-designed training and mentoring. [20:35:47] and if you want to compare tutorials with an alternative approach that we know actually works, then the conditions should be 1) control 2) tutorial invite 3) teahouse invite [20:35:48] Like the Teahouse or this tutorial. [20:35:52] My preference would be just to go with a control of no-template at this point, and if it does show a benefit, there will be plenty of opportunity to do more. [20:36:36] neilpquinn we already know the Teahouse works [20:37:06] J-Mo: yes, for sure, but it allows us to zone in a little more precisely on what about it works. [20:37:54] agree that it would be useful to study multiple conditions with multiple templates. still think the first round should be Tutorial vs. control. [20:39:00] J-Mo: and that's why it seems less interesting to test against the Teahouse. It's been tested and works, whereas it sounds like welcome templates have never actually been tested even though we have a strong suspicion that they don't. [20:39:24] anyway, it's up to you two. Fair point that you don't need to do everything first time around :) [20:41:10] ragesoss can make the final call. assuming there's a "canonical" welcome template to test against, shouldn't be difficult to implement that condition. main concern is that we get embroiled in an argument with folks who hate welcome bots if we go that route. [20:41:25] of course there's not. [20:41:40] In fact, ragesoss, you might want to remove the word "welcome" from the template title ;) [20:41:49] I mean, kind of there's {{welcome}} [20:42:01] Template:Training_modules_invite? [20:42:26] Why? I mean, it *is* a welcome bot. I do want to test whether the right content in an automated welcome is a good thing. [20:42:33] yeah, sure. [20:42:53] I guess it removes a tiny bit of potential for bikeshedding and sidetracking. [20:43:19] but the concept of a welcome bot is already the context for this in the on-wiki discussion. [20:43:19] and there's a lot of potential for bikeshedding and sidetracking: https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/HostBot_2 [20:44:30] holy shit. [20:44:31] smh [20:44:49] didn't realize you'd had that much trouble getting the TH task approved. [20:47:00] I try not to think about it [20:47:41] almost everyone in the discussion was GF, though [20:48:29] ish. [20:51:06] ragesoss when you get a chance, can you start updating this page? https://meta.wikimedia.org/wiki/Research:Training_for_new_users [20:51:31] I think I have everything i need to start updating the code [20:51:41] yeah, sure. [20:51:45] will try to do that tonight.