[00:13:05] DarTar, warning, I did what you suggested and added an entire slide for content consumption [00:16:18] Ironholds: thanks, sorry I didn’t get to respond earlier [00:16:44] after talking to toby I think we’re going to keep the Q2 goals section pretty short [00:17:17] i.e. we need to be able to articulate what these research areas feed into, even if we’re not going to have individual slides for each of them [00:17:20] DarTar, okay, just treat the slide as "if they ask for more detail, this is what I'm gonna say" [00:17:26] exactly [00:17:48] oh, and we’re also going to ditch names/initials [00:19:03] Ironholds: I think we need to refer to UC in the slide, not necessarily LUCID [00:19:18] DarTar, aww [00:19:26] but socialising terminology! [00:19:39] well, LUCID is for interim UC [00:19:43] nope [00:19:46] no? [00:19:55] LUCID is for the permanent solution. L is limited, as in limited-duration-the-cookies-expire. [00:19:55] what does the acronym stand for again? [00:20:09] ah right [00:20:15] When we go public with the consultation I don't want people throwing "unique clients" around because it's going to give the wrong impression of the privacy implications [00:20:21] hence using LUCID wherever the topic comes up [00:20:31] * Ironholds does have people skills, see? [00:20:48] we don’t need to settle the terminology question now, but yes I see your point [00:20:57] * DarTar nods [00:21:12] so plan is, kill 18 but use LUCID if it comes up? [00:21:23] I’ll create an appendix section [00:21:29] let me do it right now [00:22:05] {{done}} [00:22:37] can I ask you one more thing? replace the names in the slide with a line explaining what this is about [00:22:41] like a definition list [00:23:03] gotcha [00:23:10] (no rush, btw, as long as we have this by tomorrow afternoon PT) [00:23:30] our dry run with Howie/Erik is at 4pm PT [00:23:43] thx dude [00:24:08] totally [00:25:22] Ironholds: also, Lila will appreciate flagging some of these items as goals vs stretch goals (if somebody asks) [00:25:31] DarTar, yeah, I was actually about to do that [00:25:40] Hey Ironholds, I'm missing what you are talking about in slide 18 [00:26:27] argh [00:26:30] DarTar deleted slides! [00:26:36] no I didn't [00:26:40] I just moved them around [00:26:43] 24. Slide 24. [00:26:48] DarTar, same difference for these purposes ;p [00:27:17] halfak: we’re preparing a couple of extra slides for the appendix [00:27:32] articulating items listed under topical research [00:28:12] if you have bandwidth to do the same for WikiCredit before EOD tomorrow, that would be super-useful [00:28:28] No problem DarTar. [00:28:32] will do [00:28:43] I don’t expect us to go through the appendix during the presentation, but if people have questions we can bring up these slides [00:28:52] halfak: cool, thanks [00:29:03] DarTar, do you want a separate slide? [00:29:09] yes please [00:29:12] Sure [00:29:15] clone the one Ironholds is creating [00:29:24] ideal format: a definition list [00:29:56] I think we're abusing

[00:29:57] a label in bold followed by a more human-readable description of the goal [00:30:17] yeah, and Tufte would kill me [00:30:36] :P Tufte. [00:31:29] lzia, ewulczyn: I’m probably going to ask you to do the same ^^ [00:31:40] - go to the draft of the quarterly deck [00:31:52] - scroll down to slide 24 [00:32:29] - clone it into a new slide and add some context about the projects listed under topical research [00:32:48] DarTar, okay, thoughts on current construction of content consumption? [00:33:04] these are non-commital slides, but we can use them if people have questions about individual research topics [00:33:07] lzia, while you're here, you're first on my list of thank-yous in the paper I submitted with Scott. I hope that's cool :) [00:33:08] Ironholds: checking [00:34:16] Ironholds: looks good, I find the desc and label of the 4th item a bit mismatched [00:35:06] oh and BrE will be autoreplaced at sight from the deck [00:35:22] DarTar, you've gone native. scum! scum! [00:35:35] what do you find mismatched? [00:35:38] unless it’s Standard Scottish English of course [00:36:02] I can rewrite it in that [00:36:11] “Reader rhythms and behavioural patterns” vs its description [00:36:54] Ironholds: I should buy you a virtual beer to hear your thoughts about the Scottish vote, I’ve been talking to a ton of British friends about it [00:36:55] DarTar, BrE? [00:37:07] “standardised” WAT? [00:37:42] Ironholds: excellent, that’s exactly what we need [00:38:31] bonny unique client identifier (BUCID) [00:38:44] DarTar, done [00:38:47] * halfak has no idea what you guys are talking about [00:38:54] halfak, British English [00:38:59] so I've now rewritten the entire thing in Scots [00:39:00] halfak: lulz, check out slide 24 [00:39:05] "Proposed Q2 goals: Whut we be daen wit content consumption" [00:40:35] halfak: we can totally have a value-added metrics slide in Standard Minnesotan [00:41:11] there's actually probably a substantial overlap between scots and standard minnesotan [00:42:50] It's hard because standard MN is mostly proper with longer vowels. [00:44:33] * halfak struggles [00:44:43] halfak, make everything into a question [00:44:46] lol [00:44:50] and instead of "stretch goal" "nothing to complain about" [00:46:28] Whenever I start typing, the academic homunculus in my head complains. [00:51:27] * Ironholds blinks [00:51:45] knowing what a homunculus is (literally rather than alegorically) that sounds distinctly unhygenic [00:52:30] DarTar, so your slides will be in Italian, right? Il mio aeroscivolante � pieno di anguille [00:52:44] Ironholds, https://en.wikipedia.org/wiki/Homunculus#Modern_science [00:53:03] oh, interesting [00:53:07] it's like a biological mercator [00:53:27] Ironholds: I don’t know what you just machine translated but that sounds brilliant [00:53:41] DarTar, "my hovercraft is full of eels"? [00:53:45] ha ha [00:53:49] and who said I used machines? I might just suck at Italian [00:53:59] word [00:56:10] oi! [00:56:15] My Italian is as good as your BrEng. [00:56:21] * Ironholds mexican standoffs this thread. [01:49:44] Ironholds: I was looking for the cause of the spike in visualizations for https://pt.wikipedia.org/wiki/Cr%C3%ADticas_%C3%A0_Rede_Globo [01:50:05] Helder, aha, in PVs? [01:50:22] I can look into that :) [01:50:23] Ironholds: the original was originally posted on https://www.facebook.com/groups/grupowikipediapt/permalink/744199602319021/ [01:50:44] btw: it seems the stats are down: http://stats.grok.se/pt/latest90/Cr%C3%ADticas_%C3%A0_Rede_Globo [01:51:05] uh-oh. Henrik's issue but all of our problems :( [01:51:13] I'll take a look at the raw data and see what I can see. [01:51:20] do you know when the pageviews peaked? What day(s)? [01:51:24] that way the query will run faster [01:52:00] ~20 and 21 of this month [01:52:36] (or maybe 19 and 20?) [01:52:58] I'll go for the 20th to be safe then :D [01:54:46] thanks [01:57:37] Helder, okay, query running :). [01:58:07] looks like it's gonna be a pretty quick run, too! [01:58:18] god bless projects with < data than enwiki. [01:58:21] Everything feels so fast [01:59:09] haha... at least something good for us :-) [01:59:39] seriously, dude [01:59:44] fastest hive query over a day of data EVER [01:59:52] Ironholds: btw: what kind of data do you have to do this queries? [02:00:10] of course that looks like it's because it didn't retrieve any rows. Interesting. [02:00:10] a table with some fields? [02:00:19] yeah. All of the webrequests in a single table. [02:00:42] what is the format of a typical row of such a table?] [02:00:56] https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive#Webrequest_Table.28s.29 boop :) [02:01:22] thanks [02:01:42] okay, let's try with the unescaped url... [02:08:23] Helder, okay, I'm still seeing no rows. This is weird. Lemme run some diagnostic queries. [02:09:01] weird [02:09:30] eh, I probably just mucked up the query :D [02:09:36] or it's the special characters. URL encoding is weird. [02:09:44] oh, wait, I know..bah. [02:09:45] I know what it is. [02:37:34] Helder, okay, that page has...two. pageviews. in the entire period. [02:37:36] I'm really confused. [02:38:07] hrm [02:38:53] I'll keep fiddling with it [03:00:27] Helder, this is baffling. I'm really not seeing many requests for that page at all. [03:04:05] =/ [03:05:00] Ironholds: were all your tests for the same day? [03:05:28] maybe once the tool is running again, we can get the correct day [03:08:38] Helder, that may be it [03:08:41] or it may just be URL encoding issues [03:08:48] there are 30453453566470 ways to display that title :D [11:40:00] Ironholds, halfak, good morning. [11:41:16] Ironholds, sorry I missed your message yesterday about acknowledgements. I'm not sure which work the paper is about. but thanks for thanking. ;-) [12:41:11] halfak, why in god's name are you awake? [12:53:14] :P Was making coffee. [12:54:11] Oh! Aggregate function in the WHERE clause. [12:57:02] so I solve for this with..oh god, not a subquery :( [12:57:33] No worries. We have good options. [12:59:33] such as? [13:00:18] Email sent. [13:00:24] set a variable or perform a subquery. [13:00:34] sensible! Danke [13:00:46] I've never actually set SQL variables before. I should. [13:07:26] halfak, the subquery solution works. Yer great :D [13:07:33] \o/ [13:07:38] you know what this means, by the way? [13:07:54] that was the last necessary component in daily, geolocated, automatically gathered PV data from the sampled log files, streamed into MySQL [13:08:12] woot! [13:08:19] I'd already worked out how to fire it in there, it was just handling "okay, some of the last day's requests will be in the day before's files, so..fuck." [13:08:47] answer: extract from table, remove rows, run query over [data file matching max(timestamp)], smush the two together, fire in again [13:13:58] hey HenriqueCrang :-) [13:19:15] halfak, re table key problems, a hypothetical [13:19:32] if I was, and this is obviously just a random example, firing this query at every single database in the store, one after another [13:19:39] could this explain part of the problem? ;p [13:34:28] Ironholds, I don't think so. Then again, I don't know if I have ever heard of that error before. [13:34:39] FWIW, I fire a query against all the DBs regularly. [13:34:54] I have a helped script that will run queries in quick succession over the DBs I specify. [13:34:58] *helper [13:35:37] yeah, ditto [13:35:43] * Ironholds headscratches [13:35:48] eh, I'll see if Otto has any ideas! [13:38:51] You might ping springle too. [13:39:01] He's most likely to have insights. [13:44:33] makes sense [14:10:07] * halfak would like to have a talk with whoever thought that replacing NULLs with zeros in INT columns was a good idea. [15:12:24] leila, re paper; I turned my wikimania presentation about geolocation into a CHI note with Scott Hale :) [15:12:36] cool. [15:12:39] (there was another paper with Heather/Brent/Dave/etc but) [15:12:52] got it. good luck with it. :-) [15:13:01] I'll need it. Kind of terrified. [15:13:26] nah! don't be! either way is a good experience. [15:31:42] leila, absolutely! [15:31:48] And hopefully one I will get a chance to replicate [15:31:55] but it is still very new to me, and so for that reason I'm nervous [15:34:14] halfak, you know the dude from the conference with the many, many geo questions? [15:34:23] yeah [15:34:39] Or rather, I remember who you are talking about. I wouldn't say I know him. [15:34:46] Conversation we just had "why are there tons of links in [page]?" "well, you're asking for the DOM, and it includes the highly visible template at the bottom full of links" "what's a DOM?" [15:35:08] I...I don't...*waves hands* [15:36:20] It *is* kind of hard to google. [15:36:57] it's at least better than "some of these IPs have colons in them, what the fuck" [15:36:59] THAT was fun. [15:37:07] heh [15:40:02] goddamnit R [15:40:03] > as.POSIXct("2007-10-12 00:14:58") < "2007-01-02 00:00:00" [15:40:03] [1] TRUE [15:40:06] WHY [15:40:37] Why even allow the comparison if you're going to do it so stupidly. [15:40:49] * Ironholds headscratches [15:40:50] what? [15:41:09] Note that a date in Oct. 2007 is LESS THAN a date in Jan. 2007. [15:41:18] Unless I convert both explicitly. [15:41:46] oh, gotcha [15:41:55] so it's doing logical comparisons between a date and a character?! [15:42:07] So it is performing some coercion on the back-end. [15:42:25] oh, I see what's happening [15:42:35] ...bahaha. Oh gods it's so much worse than you think. [15:42:41] No it isn't. [15:42:44] I think it is terrible. [15:42:52] no, it's worse. [15:43:01] > 1 < "turnip" [15:43:02] [1] TRUE [15:43:44] POSIX classes boil down to a count of seconds from 1 January 1970. IOW, a numeric value. I bet this boils down not to "R handles times stupidly" but "R handles character/numeric comparisons stupidly, and is casting the POSIX timestamp as a numeric value to compare" [15:43:57] it's allowing logical comparisons of a numeric value AND THE STRING TURNIP. GAH. [15:44:10] Yes. That's what I was complaining about. [15:44:13] ahh [15:44:15] It should do a sane coercion. [15:44:20] Date works that way [15:44:26] I think the coercion is actually sensible. I mean, on the POSIX side. [15:44:49] Na. POSIX should try to convert whatever is on the Right to POSIX before making the comparison. [15:44:52] it's the fact that it allows for numeric/character comparisons at all. I mean, what the hell is it doing with "turnip" to make it comparable?! [15:44:57] This is standard practice. [15:45:04] eh, fair. This is more your domain than mine [15:45:11] I may email Adam and show him and just go "explain" [15:45:31] > as.Date("2007-10-01") < "2007-01-01" [15:45:31] [1] FALSE [15:45:48] > as.POSIXct("2007-10-01") < "2007-01-01" [15:45:48] [1] TRUE [15:45:50] AHHHH [15:45:57] * Ironholds patpats [15:46:01] use lubridate. It's sane. r. [15:47:43] * Ironholds is now going to have to read grammar.h to find out what the fuck it's doing to compare chars and ints [15:47:48] you see what you've done to me? [15:47:56] Don't do that. [15:47:57] :P [15:48:04] Also lubridate... will scope out. [15:48:38] wait, you've not used it? [15:48:42] aw man. You're in for a real treat. [15:49:07] it does tz conversion with tzdata compatible tz names [15:49:22] and you can set individual components of a date or time, like month(x) <- 1, or hour(x) <- 12 [15:49:28] UTC is the only TZ name I care about. [15:49:32] and it's not only compatible with Date and POSIX classes [15:49:44] it's also compatible with libraries like zoo and similar time-series/etc packages [15:50:03] outside of all of that it also just makes the syntax consistent [15:50:06] Oh. That's nice. So I can use the libridate as an axis in ggplot>? [15:50:32] yup [15:50:41] oh, also you can extract "what day of the week was this?" which is endlessly useful [15:51:12] and it adds %within% as a reserved phrase, to do "is this date within this period" [15:51:18] http://cran.r-project.org/web/packages/lubridate/lubridate.pdf really do recommend checking out [15:52:57] https://commons.wikimedia.org/wiki/File:First_namespace_edit.density.by_attached_method.enwiki.svg [15:53:02] https://commons.wikimedia.org/wiki/File:First_namespace_edit.density.by_registration_epoch.svg [15:53:48] Pretty graphs of the time between registration and first edit in a namespace: ^ [15:54:21] Say Ironholds. It looks like we aren't doing research group this week. Is that right? [15:54:44] we're not? aw. I wanted to make you all look at my horrifying code. [15:54:58] There ain't no if statement like a triple-nested if statement! [15:56:23] heh. I think our Quarterly has smashed our chances at having one. [15:56:27] :( [15:56:55] Do you think you'd like to use the whole hour for WMFutils? [15:57:00] (next week) [16:30:14] halfak, probably not! [16:30:17] maybe 30 mins? [16:30:24] Just because demoing the hive stuff is never short [16:30:45] Hokay. Ima take some time for socio-materiality stuffs then [16:52:43] J-Mo, hit me up about the geolocation! :D [16:54:56] will do [17:01:35] dammit [17:01:40] I really want to do the tutorial this week. Bah. [17:27:02] For those interested in namespace numbers, see https://en.wikipedia.org/wiki/Wikipedia:Namespace [17:27:50] Generally, I agree with gathering the namespace *name* when presenting data, but it's actually a substantial amount of trouble to do the conversion when plotting, so I avoid it until the final writeup. [17:28:03] Ironholds, do you have a nice namespace solver for WMFUtils? [17:28:13] halfak, define solver? [17:28:22] you want numeric values to strings or strings to numeric values? [17:28:25] I give it a namespace name and it tells me the ID. [17:28:31] I give it an ID and it tells me the name. [17:28:39] ooh. no. But I could. [17:28:44] That'd be really trivial to build, too [17:28:48] I already know how to retrieve them [17:28:48] http://pythonhosted.org/mediawiki-utilities/lib/title.html#mw-lib-title [17:29:12] That's how I solved it. The Parser can be constructed with an API session. [17:29:16] Or manually. [17:30:38] what I'll probably do is save it as an RData file and create an associated function that can be run to regenerate said file [17:30:49] so, a solver, a dataset for the solver, a generator for the dataset [17:30:49] Regretfully, namespace names are complicated with "cannonical" names, aliases and case sensitivities. [17:31:16] That way it can run on stat2 and other non-net-connected machines [17:31:32] dammit. There was another dataset I was going to generate as well and I can't remember what it was. Ah well, it'll come to me. Thanks for the idea! [17:31:38] +1. You could make a "parser" constructable directly from the RData or via an API call. [17:31:56] yup [17:32:11] if there's a decent way of checking net connectivity. I guess I can just add it as an option [17:32:23] if(use_API){ [17:32:36] namespace_names <- NamespaceGenerator() [17:32:47] } else { load("namespace_names.RData") [17:33:04] it's probably okay to leave it up to the user to work out if they have internet access [17:33:06] ;p [17:33:45] Ironholds, I'd leave it up to the dev. See my "from_..." functions. [17:33:57] Got a dump parser? Give it to from_dump() [17:34:13] Got an API session? Give it to from_api() [17:34:20] * Ironholds nods [17:34:41] Got the raw JSON of a call to siteinfo (could have been stored in a file)? Give it to from_si_doc() [17:37:51] yep [17:38:04] ack! I remembered the other dataset [17:38:15] sitematrix data. Specifically, primary languages/dialects and local names for each project [17:38:19] to make API queries convenient [17:38:35] Oh yeah. That's the one that populates staging.wiki_info [17:38:37] well, the URLs make API queries convenient. But the languages/dialects is nice info [17:38:38] yep [17:38:54] and now I know what I'm spending my remaining energy today on. cool! Thanks :) [17:39:17] godspeed sir. [17:54:06] hey Ironholds [17:54:22] can you start filling out https://meta.wikimedia.org/wiki/Research:Page_view as we said yesterday? [18:01:58] Ironholds, ready to sqoop? [18:02:18] leila, sqoop [18:02:34] halfak: I think Ironholds is off to a doctor’s appointment [18:02:41] Gotcha. Thanks. [18:02:57] and if he’s not, he should not sqoop but work on something I asked him [18:26:35] DarTar, about to head off, yeah [18:26:40] and after that I'll be up writing up page view [18:26:49] cool, danke [18:26:55] would you like the background section in there as well with a warning? [18:27:04] "this is our best understanding, can and will change, etcetcetc" [18:27:17] add a {{draft}} [18:27:18] ? [18:27:35] yup [19:45:14] DarTar & Ironholds. Want to do a make-up session with ottomata on sqoop? [19:45:32] halfak: I’d love to, or have you update us [19:45:36] whichever works best [19:45:57] Shall I put something on the schedule for next week? [19:46:34] sure, I don’t yet know what’s going to happen with the analytics offsite [19:47:12] Meh. that's still a couple weeks out. [19:47:23] I think it would be a good idea to play with scoop a bit before the offsite. [19:47:35] We could spend some time at the offsite talking about recurring imports then. [19:47:51] Oh wait... offsite or onsite? [19:47:57] site == SF office? [20:34:16] STAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAF [20:34:30] DarTar, IH|docs, leila ^ [20:34:36] tnegrin, ^ [20:34:48] halfak: yeah, we’re trying to pull tnegrin out of another meeting [20:34:53] um [20:34:59] halfak: are you having a problem? [20:35:00] Hey Pine! :) [20:35:15] heh. Trying to get these guys to come to the staff meeting. [20:36:12] halfak: announce that there will be free pizza and beer. That will motivate people to attend. [20:36:31] Problem: not colocated. [20:36:38] Otherwise, yes. [20:36:55] Announce that the meeting involves an all-expenses-paid trip to Bali. [20:40:52] hi all [20:40:55] hey Pine [20:41:09] Hi tnegrin [20:42:36] so on the deck -- the performance to Q1 goals looks good [20:43:45] slide 16 -- can we add something other than "augment team capacity" as the justification? [20:44:14] I think there are many good reasons to do this [20:53:20] can anyone point me to the MediaWiki page for the "embed charts in wiki pages" extension? DarTar, tnegrin, halfak? [20:54:08] J-Mo, are you looking for the work with Vega -- a language for describing a chart? [20:54:27] This was a project that milimetric worked on at the Zurich Hackathon. [20:54:35] It was more a proof of concept than a working thing. [20:54:58] I don't think so. it was mentioned in a meeting this morning with tnegrin and kevinator. Creates charts on wiki pages, pulls from uploaded csv datafiles. Sounded like it was operational. [20:55:00] https://www.mediawiki.org/wiki/Extension:Graph/Demo [20:55:03] That might be right. [20:55:21] halfak / JMo, yep, Yuri took my proof of concept and made it legit [20:55:28] :) [20:55:59] nice :) thanks milimetric and halfak [20:58:43] J-Mo: I have a question if you have a sec. And by the way, Halfak and I are meeting this evening to discuss that networking viz. [20:59:53] hi Pine. I'm ready when you are (and I really wish I had capacity to meet re: the networking viz project… ) [21:00:40] J-Mo: just wondering when we could schedule the October regular meetup so you can actually join us. Feel free to email me in private. [21:01:33] definitely 10/14 will work. I may be out of town on 10/7. [21:01:42] ok thanks [21:06:55] bleeeeh [21:07:17] Hey dude. [21:07:20] JOIN US. [21:07:25] Ironholds, ^ [21:08:15] oh, you're still going? [21:08:16] awesome! [21:22:17] halfak, I resolved "why R does that dumb shit" btw [21:22:44] "character, complex, numeric, integer, logical and raw" is the conversion order for vectors of different types. [21:23:05] For all types? [21:23:32] I assume that POSIX are numeric/integer values when you boil it down; I mean, it'd make sense given what they're representing (an int value of seconds) [21:23:32] I mean, even if POSIX did character conversion, the comparison would have worked. [21:23:56] how it's handled lexicographically I don't know, though [21:24:09] I mean, it explains 1 < "turnip". But sorting different character strings numerically? Ew. [21:24:24] > as.character(as.POSIXct("2010-01-01")) [21:24:24] [1] "2010-01-01" [21:24:37] That's how MySQL does date comparison (or at least how it used to) [21:24:44] huh! [21:25:52] > as.character(as.POSIXct("2007-10-12 00:14:58")) < "2007-01-02 00:00:00" [21:25:53] [1] FALSE [21:26:04] When I explicitly convert to chars, it works. [21:26:07] HUH [21:26:09] wat R [21:26:10] wat [21:26:16] then it's not doing what the documentation says it should. [21:26:28] * Ironholds blinks [21:26:37] this is the dumbest language I've ever been involved in and goddammit I SPEAK ENGLISH. [21:26:44] lool [21:26:59] Still could get into PHP [21:29:16] fair ;p [21:29:18] okay, back in a tick [22:15:40] DarTar, https://meta.wikimedia.org/wiki/Research:Page_view [22:30:51] hey guys, we’re switching rooms, give us 2 mins [22:34:29] halfak, Ironholds - are you guys around? [22:34:44] yup. Sorry. [22:39:01] DarTar: https://etherpad.wikimedia.org/p/WRN201409 [22:39:04] does not work for me [22:39:16] tries to load, blinks and then gives an error [22:39:30] are there etherpad problems over there? [22:45:54]