[21:41:00] Nettrom halfak: is there a top-line 'accuracy' stat for the ORES wp10 model? Like, what portion of articles it predicts the correct rating for? [21:44:58] ragesoss: there should be a way to ask the API for info about the wp10 model, I don’t remember where to find it though… halfak? [21:59:40] https://ores.wmflabs.org/v2/scores/enwiki/wp10/?model_info [21:59:45] ragesoss, Nettrom ^ [22:00:45] halfak: thanks, bookmarked it for future reference :) [22:04:37] thanks. tons of numbers there that I don't know how to interpret. But one related question: how does the distribution of quality ratings in the training set — roughly even numbers for each class — affect the model? As in, the actual distribution of article quality is very different from that. Does that matter at all? [22:19:04] mako: I'm reading the Revisiting RAD paper. Why were you only able to do get Wikia data through April 2010? [22:19:59] ragesoss: the short answer is that wikia only made automatic public exports of /verything/ up through mid-2010 [22:20:45] ragesoss: after that, you sort of have to request them. and there were export problems. and they did a series of mass-deletions of wikis. the result is that don't actually know that we have everything for any period afer that [22:21:18] ragesoss: we absolutely can get data on wikia wikis from after 2010 [22:21:34] ragesoss: but not the kind of comprehensive data you could get before [22:22:11] mako: I see. What is the distribution of Wikia wiki size? Something like Pareto? [22:23:46] ragesoss: i'm sure depends on what you mean by size.. but "some like" in the sense that it will be long-tail power law distribution or something similar in shape for most measures of size [22:24:16] ragesoss: large majority are tiny, a few are bigish, a tiny number are totally massive [22:25:12] right. and these 740 are the top 1%, which themselves have a few that are pretty big and most fairly small. [22:25:13] ragesoss: often what we've done is selected lists of wikis from the 2010 list because we believe it's the closest thing to a "population" we have to sample from. then we try to chase down more recent exports [22:25:21] ragesoss: that's correct [22:25:36] ragesoss: we have a robustness check in the paper that addresses an potential issue with this [22:26:53] ragesoss: which is that our/their model is about users and most of the users will come from the biggest communities. so it's still about the average user experiences but it is kind of a like a less extreme version of the "problem" with RAD: it tends to give much more weight to the biggies [22:27:55] ragesoss: it might be that this is just a problem that big wikis have [22:27:56] mako: related: the bigger wikis are probably also the earlier ones, correct? [22:28:37] ragesoss: but our robustness check weights every wiki equally and finds, IIRC, an even stronger effect [22:29:03] it's not clear that equal is the "right" way to weight things either, but it seems instructive [22:29:13] ragesoss: i don't see any reason to assume that the big wikis are earlier, but you might be right [22:29:48] I mean, I guess there are probably major exceptions to that with Wikia, because of wikis built around some media property that was released at a certain point in time. [22:30:21] ragesoss: possibly. but many big wikia wikis are about media, games, etc, that were released later. i think size is probably mostly a proxy for underlying interest in the topic [22:30:22] Basically, I'm really curious about whether there is *any* explanatory value to calendar time. [22:30:58] ragesoss: oh, we actually controlled for that. but we controlled for it in a way that makes it hard to answer that question. [22:31:21] like, I believe that 'rise of facebook, changing norms of internet' don't significantly explain RAD. [22:31:34] ragesoss: like, we can be "pretty sure" we took care of it, but it's harder to charactertize the nature of what we took care of [22:31:38] it's possible to look [22:31:44] we get should get nate into this channel :) [22:32:01] yeah. [22:32:51] If you can learn anything about the broader context of the internet from the this same data, even if the effect is small, that might be super interesting. [22:32:51] ragesoss: i just invited him (like IRL) and i'll paste the backlog to him [22:33:33] ragesoss: we definitely did something that would/might speak to that, but i don't recall the answer :) [22:33:46] ragesoss: lets see what nate (groceryheist) says [22:37:28] o/ [22:37:36] I heard people were talking about my project in here [22:37:47] mako shared the log [22:37:48] just me and mako. [22:38:07] i don't really think big wikis are earlier [22:38:20] i can find that out pretty easily [22:38:34] you can too once I finish publishing the archive data [22:38:52] ragesoss: i gotta run to a meeting, you're in good hands :) [22:40:09] ok ragesoss [22:40:21] like actual answers from the person who did the work rather than my hazy second-hand recollections ;) [22:40:43] :-) [22:40:52] so we can't answer the overall time question, since we have seperate time variables for each wiki [22:40:59] there is not any global time variable [22:43:17] we could fit a hierarchical model with a global time variable [22:43:26] that would be another study though [22:43:37] groceryheist: the lead graph that shows rise-and-decline... do you have a sense for what the Wikipedia curve looks like in comparison? [22:44:41] like, I infer that in units of std dev, that means we're actually looking at exponential growth in the first years, just like on Wikipedia? [22:44:56] yeah 'Screenshot from 2018-03-27 15-44-09.png' [22:44:58] oops [22:45:05] 'Screenshot from 2018-03-27 15-44-09.png' [22:45:07] but the peak comes at year 4, which is several years before the peak for Wikipedia. [22:45:10] damn having trouble [22:45:22] https://teblunthuis.cc/outgoing/rise_decline_wp.png [22:45:23] ok [22:45:50] you are right about the stdev units [22:45:56] the timescale is different though [22:45:58] but if it was graphed the same way you do, it would look more similar in shape? [22:46:05] oh i see [22:46:10] yeah I should build that graph [22:46:23] good question [22:47:28] so the timescale is different [22:47:44] one issue with the wikia graph is that different wikis are on different timescales [22:48:10] which I think makes the graph smoother than it would be if they were all on the same timescale [22:48:25] just different starting points, corresponding to when that wiki started, right? [22:48:35] that's what I have done [22:48:47] but some wikis might peak in 2 years and others in 6 [22:49:09] yeah. would love to see a plot where every wiki gets its own line. [22:49:16] it looks awful [22:49:23] total hairball [22:49:35] to see the variation in how long it takes to reach inflection. [22:49:48] there could be other ways of doing that [22:50:04] like trying to fit a logistic curve to each and then aligning the inflection points [22:50:17] but I didn't do that yet :0 [22:50:20] :) [22:51:23] I think 'how do wikis that started at different points in time behave differently' is a very interesting question. [22:51:24] one of the things about doing a replication study is minimizing the deviation from the original analysis [22:51:53] yeah, this is a different study we're talking about now. [22:51:54] ;) [22:52:10] yeah it could be, what do you think? [22:53:00] well, I guess it's hard to convince me that the rise to dominance of Facebook didn't matter at all. [22:53:36] I agree [22:54:06] I would claim the analysis suggests that Facebook can't be the whole explination [22:54:08] among other changes in the internet, some more along the lines of changing infrastructure (more bandwidth, more mobile), others around changing expecations for user experience. [22:54:18] but i would be astonished if it had no influence [22:54:33] yeah [22:55:00] like, that's basically the hypothesis that drove Wikimedia strategy for a long while. [22:55:33] the hypothesis that changing expecations of user experience and broader technological ecology was the concern? [22:55:34] and was the basis for focusing so heavily on, for example, Visual Editor. [22:55:56] the combination of that, plus Facebook slurping up the cognitive surplus. [22:56:20] i think that the UX stuff is probably mainly a concern upstream in the pipeline [22:57:15] by the time people start editing it's still an issue, but stuff like norms and rejection matter too [22:57:48] I don't know about cognative surplus [22:57:57] that seems hard to measure [22:58:09] I think part of this was down to the individuals involved, and their higher level strategic concerns. Like, Erik Moeller was setting much of the direction, and I think the existential threat of Facebook to the goals of the free culture movement were mixed in there. [22:59:00] which is part of the same reason why I am really interested in these questions of how peer production may have been affected by calendar time. [22:59:15] anything that might help us destroy facebook. [23:01:12] cool well I can fit a hierarchical model and we can find out [23:02:08] I think if you can find anything that is tied to calendar time, that could be a super interesting result that you could use as the starting point for many other stories / questions. [23:02:35] part of the challenge is that time is related to everything [23:02:46] so it's pretty amenable to post-hoc explination [23:03:02] so I'd kind of like to have more of an apriori hypothesis. [23:03:02] yeah, for sure. [23:03:34] it's also hard like how much to attribute to facebook, how much to apple, how much to broadband, netflix and so on [23:03:44] but if you can say 'online community dynamics have been changing in this way over time', that's pretty provocative. [23:04:16] yeah i think so too