[09:42:28] YuviPanda: was just composing a reply re: user-agents [09:42:43] oh, that was a while ago [09:42:47] I think [09:43:22] eeep. you're right. I had just noticed it. [09:43:25] ori-l: and we're logging those separately now anyway. (in the app) [09:43:35] separate platform, device, etc fields [09:43:37] easier to analyse :) [09:43:47] ori-l: also https://gist.github.com/yuvipanda/5078476 [09:43:49] ok, good. that's a relief. [09:44:04] ori-l: yeah. Mobile web logs their UAs [09:44:38] (note about how my SQL is pretty horrible should preface that script) [09:44:42] window.navigator has more granular properties [09:45:00] we reallllllly have to start getting numbers from navigationTiming [09:45:43] :D [09:46:04] and the app doesn't actually do persistent logging yet (network errors just fail silently) [09:46:07] should to at some point of time [09:48:01] yeah, that'd be neat [09:48:10] yeah. [09:48:15] and would need to add client times [09:48:25] too many things to do, only me to do it :( [09:50:45] you no longer need to import Template or to define render [09:51:01] also, 'unicodecsv'? [09:51:27] ori-l: yeah, default python csv module chokes on csv [09:51:42] ori-l: I'm passing my SQL through Jinja [09:51:48] (to avoid repetition) [09:51:55] so will need to import template / render [09:52:04] you crazy bastard [09:52:13] :D [09:52:14] that's kinda neat, actually [09:52:20] indeed [09:52:29] I don't want to do a find and replace if we push a new version of the schema [09:52:33] or decide 'oh, 30 days, not 7' [09:52:48] or any such thing [09:53:13] ori-l: once this gets big enough, I should refeactor that further and write up template based generators or something [09:53:20] but only if it is going to be simpler than th eSQL itself [09:53:58] ori-l: i'm just too lazy to put in the effort to learn SQLAlchemy [09:54:00] this does look like it would be nicer to do with mongo [09:54:04] true [09:54:21] ori-l: I was looking at their aggregation framework yesterday [09:54:24] but i think i would do something even more drastic and just load the entire 7 day event set into memory and analyze it in python [09:54:41] Ironholds suggested the same thing [09:54:48] but for some reason even thinking of it makes my skin crawl [09:54:57] :P [09:55:05] ori-l: what happens when it is 30 day than 7? [09:55:17] (30 is probably what it will end up as, it is 7 now because we don't have that much data yet) [09:56:03] well, my hunch is this -- [09:57:28] (sorry, having trouble formulating the thought; just a moment.) [09:57:41] haha :) [09:59:35] i don't have a ton of experience working with analytic data, but my experience in E3 has been that the hard scaleability problems are mostly wetwear-related [10:00:10] heh [10:00:16] there's an initial period of "weee, data!" but that is gradually replaced with "wtf does it all mean, anyway" [10:00:41] I am pretty sure I'm in the 'weee data' period now. [10:00:46] and that takes a lot of hard thinking, asking yourself what you care about and how you can measure it in a way that helps guide further action [10:01:00] me too :P [10:01:00] right. [10:01:03] but thinking ahead [10:01:04] heh. [10:01:21] I think it is easier to be myopic now (isn't it always?) because there is absolutely nothing at all to look at now. [10:01:23] so i would prefer to express computation in the most human readable way [10:01:32] hmm [10:01:32] true [10:01:48] perhaps we should do actual computaiton in python, and keep the SQL solely for selection? [10:02:01] I'm guessing that most of the present SQL counts as selection [10:02:02] hmm [10:02:04] where to draw the line [10:02:12] does moving average count? [10:02:21] ori-l: I suppose mongodb might be more readable, since aggregate frmaework is pipelined [10:02:39] i don't think there's a hard and fast rule. and yes, re: mongo [10:02:55] but it is code in JSON [10:03:05] there is another fancy new datastore that i like and have been wanting to set up [10:03:05] I am preetty sure we could easily write it in YAML too [10:03:09] redis? [10:03:11] couchdb? [10:03:26] so mainstream [10:03:40] http://www.rethinkdb.com/api/#py [10:04:13] see aggregation / transformations / reductions in the sidebar [10:04:57] hmmm [10:04:58] nice [10:05:04] but not too different from mongo? [10:05:58] hrm, you're right [10:06:06] i took another look at the aggregation framework [10:06:35] plus you can always drop into mapreduce [10:07:55] let me try adapting one of your queries to mongo so we can compare [10:08:26] yup! [10:08:27] I'll also note that my queries are probably not the most efficient [10:08:34] I'm sure someone with better SQL skills could write a self join [10:08:39] but this is far more readable [10:08:41] http://www.mediawiki.org/wiki/Extension:EventLogging/MongoDB [10:08:44] and something that my tiny brain can understand :) [10:08:58] not sure if i shared that before [10:09:20] yes you have [10:09:36] ori-l: I've a doctor's appointment in an hour, so I should leave. [10:09:41] ori-l: email me? [10:09:42] sorry :( [10:09:59] i should have gone to bed hours ago, probably good to cut this short :P [10:10:06] ttyl, avoid malaria [10:11:11] ori-l: yes. SLEEP! [10:11:11] :) [10:11:15] ori-l: will try :) [10:11:15] cya [17:41:11] MaxSem, YuviPanda: do you know if this stats page is specifically for WLM uploads from the mobile app? https://toolserver.org/~legoktm/wlm/stats.php [17:41:19] otherwise, the stats make no sense! [17:42:11] awjr: oh yes, i tis [17:42:16] sweet thanks YuviPanda [17:42:16] awjr: I forked the original code and modified it to be mobile only [17:42:23] my toolserver account expired, so it is running on leg's [17:42:25] ahha awesome :) [17:42:25] *lego's [17:42:29] that is great, thanks! [17:42:36] :) [17:43:02] awjr: back to 100%? [17:43:48] almost YuviPanda, still feeling a little foggy and i was really run down by the end of yesterday, but the fact i was able to stay focussed and work a long day was awesome :) [17:49:22] awjr: :) get better [17:49:57] thanks YuviPanda :) [17:50:22] i have a weird habit of getting sick/injured prior to international trips... [17:50:43] but this is nothing compared to the time i herniated ad disc in my lower back the day before a 3-day trip to Austria [17:50:46] that… sucked. [17:50:50] awjr: you told me [17:50:53] yeah [17:51:00] you went through with it. braver man than I am :) [17:51:02] heh yeah, don't do that :p [17:51:08] lol i probably should not have [17:51:15] im not sure it was the smartest thing to do [17:51:26] you did come out of it alive [17:51:32] hahaha that is true [17:51:50] about 4 years ago, i broke my collar bone and then two days later went to Las Vegas for a weekend [17:51:52] that was also really dumb. [17:52:05] wow, apparently a pair of mutual friends got engaged. he proposed on top of the eiffel tower in a surprise trip. [17:52:10] ! [17:52:11] classy [17:52:47] indeed. I initially thought it was sortof a joke [17:52:51] but turns out to be actually true [17:52:53] cliched [17:53:09] but classy [17:53:09] when actually done [17:53:09] than just talked of [17:53:22] awjr: did you come out of the Vegas trip without more broken bones? [17:54:40] hehe i did YuviPanda. what i didn't know at the time though was that when i broke my collar bone, i did so in a patch of poison oak (http://en.wikipedia.org/wiki/Toxicodendron_diversilobum) and while i was in vegas, the rash staretd to appear all over my body. at first i thought i was allergic to the pain medication i was on, so i stopped taking it [17:54:47] ow [17:54:54] then i was in a helluva lot of pain, and the rash kept spreading/getting worse [17:55:03] that was no bueno [17:55:43] awjr: you should make a movie out of these [17:55:47] lol [17:55:54] too embarassing :p [17:55:56] 'sick'n travelling' [18:01:42] aude, around? [18:02:19] hi MaxSem [18:08:50] hey Maryana [18:08:57] what did you use to produce the stats you mailed out to us? [18:09:02] can you send me the SQL? [18:09:12] (Just looking to see what kinda stuff we use. I'm working on the apps dashboard) [18:09:47] sure :) which stats in particular? i use SQL sometimes and R sometimes [18:10:28] Maryana: all code! [18:10:32] I'm using Python / SQL [18:10:37] just looking for examples to pick up from [18:12:05] sure thing. i've been meaning to compile a list of useful queries, anyway. i'll try to get something to you by end of day [18:13:15] Maryana: :) [18:13:43] maybe one day i will use my long-neglected and empty github account! [18:13:44] heh [18:13:54] Maryana: I'm using something like https://gist.github.com/yuvipanda/5078476 to generate them (to be plugged into limn). Me and ori-l were thinking of using MongoDB too [18:14:13] (should make generating these kinds of things from EL trivial, rather than need every team to write their own thing) [18:14:45] \o/ [18:15:19] Maryana: It is going to be up on mobile-reportcard.wmflabs.org [18:15:30] I suppose Mobile web would want to put some stuff up there too at some point? [18:15:43] yeah, i should check in with juliusz [18:15:57] i think he was interested in working on that [18:16:08] yay data! [18:16:31] Maryana: yeah, he was asking me for source. [18:16:49] Maryana: would be simple enough to integrate EL data from mobile web into this. [18:16:55] do ask him to poke me if he starts on this :) [18:17:01] will do [18:17:04] There's a bug in the CentralNotice hack juliusz did yesterday :( [18:17:51] jdlrobson: awjr: I don't know any context behind bug 44070, but I just want to make a quick point [18:18:05] jdlrobson: awjr: Remember that load.php needs it, since they are loaded not from within the page [18:18:35] it can't rely on page context to define a module. startup module needs it (which is by design for caching) [18:18:42] oh hey, yuvipanda - have you thought at all about how to deal with deleted images in the apps? [18:19:01] Maryana: we've determined that it is impossible to efficiently do so with the currently available APIs [18:19:08] (me + brion) [18:19:09] urgh [18:19:17] Krinkle: thanks - would you mind noting that all in the bug? [18:19:17] i figured [18:19:20] and will probably sit and figure a way out at some point. [18:19:26] !b 44070 [18:19:32] one of the easier things to do is to show the user when they open the image for detail [18:19:36] but other than that it is not easy. [18:19:45] we've a couple of things in mind but they all require core changes. [18:19:52] oh, i see, so you're still showing them the image [18:19:56] in their contributions view? [18:19:59] well [18:20:05] that is inconsistent [18:20:09] if it has already 'synced' beforehand [18:20:11] it will still be tehre [18:20:16] but if not [18:20:17] it won't be [18:20:23] gotcha [18:20:26] the image's *name* would show up [18:20:35] the image itself will not [18:20:35] jarring experience, IMO [18:20:37] awjr: https://bugzilla.wikimedia.org/show_bug.cgi?id=44070#c5 [18:20:48] yeah, it's super weird on web - if your first upload is deleted, then your contributions view says "0 uploads" and the first-time user message reloads :-/ [18:21:00] thanks Krinkle [18:21:06] yuvipanda: i'm trying to think of a way to make this better, but i just don't know how [18:21:27] Maryana: are you guys using the images table to populate this? If so that has another problem we ran into [18:21:35] if someone else overwrites your image it won't show up as well [18:21:40] hmm [18:21:50] i'm not sure. would have to check w/maxsem [18:21:51] so if some guy edits your image a little bit, crops, whetver - boom, gone. [18:22:05] i imagine that would be pretty rare [18:22:15] oh, except maybe for rotation [18:22:30] rotation shouldn't happen as much anymore since there's auto-rotation [18:22:34] and some other bot work (jpeg deinterlation, etc) [18:22:36] but cropping and enhancement? can happen [18:22:45] bigger problem is when *you* edit someone else's work [18:22:50] and that shows up as your contribution [18:22:52] and not as theirs [18:22:56] geotagging happens :) [18:22:59] which IMO is bad. [18:23:01] maybe more likely for diagrams than photos but i've seen it [18:23:04] aude: geotagging doesn't modify the image [18:23:10] oh, ok [18:23:16] brion: I spend a fair amount of time commons-gnoming [18:23:19] usualyl cropping photos :) [18:23:22] another day, another thing about the wikis that makes my head hurt [18:23:25] :) [18:23:25] so... my feed is polluted [18:23:56] Maryana: do you have stats on deletions of mobile uploads? [18:23:58] New patchset: Jdlrobson; "Bug 44070: Define special page modules manually" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/52254 [18:24:17] YuviPanda: if we just had an 'original uploader' field in 'image' which we could search on in list=allimages would that be good enough? [18:24:30] brion: yeah, I'm thinking it would be. [18:24:31] yuvipanda, no, but i could [18:24:40] brion: but how would we handle deletions? [18:24:41] and moves? [18:24:43] that's a pretty easy query [18:24:45] hmmmmm [18:24:58] well moves would get re-synced at some point i guess [18:25:10] deletes…. might just disappear *puff* [18:25:22] I'm syncing, so I have offline data to be updated :P [18:25:41] brion: if eventlog had a deleted field that I could filter by original uploader... [18:26:06] hmmm [18:26:31] not sure how best to handle that. there's extra props on log items but not sure you can filter on them easily [18:26:54] meanwhile, http://osm.wmflabs.org/osm/slippymap.html ! [18:27:40] \o/ [18:27:41] wooohoo MaxSem [18:27:57] it's in ur labs drawin ur maps [18:27:58] yay [18:28:07] uh uh :D [18:28:11] New patchset: Jdlrobson; "Bug 44070: Define special page modules manually" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/52254 [18:28:15] brion: what're we going to do for maps on apps? [18:28:24] I don't want to go back to the 'easy solution' which would be Google Maps [18:28:40] i'm leaning towards using OSM [18:28:42] so far, there's only Ireland for performance and storage reasons [18:28:56] but we'll decide for sure when we get to that bridge. agiiiillllleeeeee :D [18:29:26] as long as it's easy to open something up in native maps to get directions, that's all the integration we probably need [18:32:21] oh yay http://arstechnica.com/gadgets/2013/03/verizons-galaxy-nexus-to-get-android-4-2-months-after-everyone-else/ [18:32:38] brion: :P [18:32:47] stupid verizon [18:32:48] brion: yeah [18:33:05] brion: i looked at some of the OSM Native android renderrrs. On first look none seem too stable. [18:33:08] (or mature) [18:33:39] and you pay like a hundred or more bucks per month to be treated like that? [18:33:42] webview & leaflet? ;) [18:34:04] MaxSem: seriously. i don't think i'm renewing with them, i'm thinking of moving to t-mobile when this contract's up [18:34:08] brion: fallback, but ideally I wouldn't want to [18:34:11] i can get no-contract SIMs from them [18:34:13] (go to webview) [18:34:33] yup [18:34:54] showing image tiles and moving them around isn't rocket science in theory, if we have to roll our own we can probably manage it [18:35:07] but if we can reuse existing, that's a big plus [18:35:07] brion: well, native renderers are not supposed to be showing image tiles :P [18:35:09] vector maps! [18:35:17] ooooh :) [18:35:18] I think there are some decent ones for iOS [18:35:32] nice i'll have to look that up [18:36:07] yeah :) [18:36:13] also Luis just turned up on -staff channel [18:36:18] and I'm pinginh him about h264 :D [18:37:02] whee [18:37:13] hah my invite broke on staff again [18:37:49] poor luis :) [18:37:58] "welcome aboard! here's a controversial decision to be in the middle of" [18:38:24] hahaha :D [18:38:50] brion: you don't have access to staff? [18:39:21] yeah there's some breakage with my invite bits or something. having someone look at it now [18:39:27] it's been intermittent for ages [18:40:06] brion: ok [18:41:23] New patchset: Krinkle; "Define special page modules manually" [mediawiki/extensions/MobileFrontend] (master) - https://gerrit.wikimedia.org/r/52254 [18:43:18] brion: we have not even opened the can of worms that is GPL'd code on iOS App store :P [18:43:25] hehe [18:43:42] GPLv2 *I think* is ok. GPLv3 would be a def no-no. [18:43:45] brion: you should put a BSD / Apache license file on the repo. [18:43:54] brion: no, GPL of any sort is a no-no on apple app store [18:43:59] it has 'additional restrictions' [18:44:05] (3 devices per user) [18:44:08] that violate GPL [18:44:17] apps have been taken down before due to this, hence precedent [18:44:24] brion: we can avoid drama by dual licensing [18:44:26] I just had an AWESOME idea for the mobile team. [18:44:26] ACT NOW! [18:44:26] :P [18:44:27] well we snuck GPLv2 in there already, we'll have to make up our minds :) [18:44:34] ragesoss: DO TELL! [18:45:10] so, we have a nice Wiktionary app, and now we have photo uploads... [18:45:30] (we *had* a nice wiktionary app, IIRC it hasn't worked for a while) [18:45:47] iOS isn't restricted to 3 devices per user… i've got more than that [18:45:53] Windows 8/RT does restrict to 5 though [18:45:59] put them together, and have people take a portrait of someone for any definition. [18:46:10] brion: IIRC Windows app store explicitly disallows GPL [18:46:29] they let me put it in dude. [18:46:36] "in the dictionary under too, there's a picture of you" [18:46:39] it says GPLv2 all over it :) [18:46:55] under foo [18:47:05] http://apps.microsoft.com/windows/en-us/app/wikipedia/6b80bf54-1a31-4651-acce-59a5b5c2b7c8 [18:47:21] brion: same with iOS [18:47:30] ragesoss: :P that sounds very urban-dictionaryish [18:47:44] ragesoss: I was thinking you were going to say 'let them upload pronunciations!' [18:48:17] brion: http://www.fsf.org/blogs/licensing/more-about-the-app-store-gpl-enforcement [18:49:29] we'll see :) [18:49:50] if we want to change the license off GPLv2 we'll have to convince tfinc [18:49:55] those articles are good things to collect tho :D [18:50:13] dual license. [18:50:15] i'm gonna go check if the water heater's fixed.... [18:50:20] heh [18:51:03] nope still no water [18:51:23] pronunciations would be more practical, I guess. :p [18:51:27] has anybody ever thought about fixing the fact that wikimediafoundation.org has no mobile-friendly version? [18:51:40] given that both our terms of use and privacy policy live there...? [18:51:42] :) [18:52:02] oh yeah [18:52:09] lemme pull up the bug on that, there might have been some reason for it [18:52:14] Maryana, there are some pages on it (like donation) that need some love to work on mobile [18:52:48] ragesoss: true [18:52:52] can't find the bug [18:52:55] maybe there is no bug [18:52:57] i thought there was :( [18:53:00] i shall make a bug! [18:53:01] "there is no bug" [18:53:03] \o/ [18:53:13] there is no bug but what we make for ourselves [18:53:17] hehehe [18:53:21] I'm sad nobody's made a reference to bug 1 in the last few months [18:53:31] well, i just filed this one: https://bugzilla.wikimedia.org/show_bug.cgi?id=45742 [18:53:51] dunno what you apps guys are linking to, but the wikipedia CC-by-SA description is really.. uh.. [18:53:55] wikipedia-ish [18:54:08] e.g., dense and hard for a newbie to understand [18:54:20] the creative commons human-readable version is way better [18:54:40] yuvipanda: what's bug 1? [18:57:23] docs suck [18:57:27] Maryana: https://bugzilla.wikimedia.org/show_bug.cgi?id=1 [18:57:49] hahaha [18:58:01] day 349583948753948573475398. our docs still suck. [18:58:04] nothing further to report [18:58:05] :D [18:58:46] "The bug is not meant to be closed. The goal is unattainable, which is why, by [18:58:46] simple logical progression, it must stay open. Like the Christian concept of [18:58:47] freedom from sin, we can only strive to fix bug 1, we can never actually fix [18:58:48] it." [18:58:52] ^ best. thing. ever. [18:59:00] lol [18:59:10] #shittimstarlingsays [18:59:16] :) [19:00:07] brion: you read through bugzilla quips anytime? [19:00:07] lotta fun [19:00:57] err [19:00:58] i meant [19:01:00] Maryana: ^ [19:01:02] good stuff [19:03:49] hehe [19:03:51] (https://bugzilla.wikimedia.org/quips.cgi?action=show) [19:05:14] "These aren't the bugs you're looking for." [19:05:56] my skin knowledge isn't very deep * Ryan_Lane puts on sunglasses one might say it was… skin deep. has to be my new favorite [19:12:22] Maryana: do we have upload stats for q2? [19:12:26] mobile photo upload, that is [19:12:36] for q2? [19:12:44] yah, oct - dec [19:13:26] no, but i can get it. it will be very unimpressive once we filter out jdlrobson :) [19:13:38] hehe [19:13:39] lol [19:13:49] * Maryana checks [19:14:39] awjr: am i correct in that we're not doing a deployment today? [19:14:44] or is it next week's that got pushed? [19:15:10] brion: we are deploying today - next week's is moved to weds [19:15:20] brion: did your i18n change for zero make it out? [19:15:28] ah ok good [19:15:28] lemme poke it [19:17:42] awjr: ok my i18n updates did _not_ go out, probably the i18n updates aren't quite recognizing them because the keys were funky [19:18:07] brion ah ok - MaxSem can you make sure brion's i18n updates get pushed with today's MF deployment? [19:18:22] thanks awjr MaxSem :D [19:18:31] awjr, query running. i'll let you know when i get #s [19:18:33] https://gerrit.wikimedia.org/r/#/c/52068/ <- is merged to master, just needs to go out with the wmf branches [19:18:40] no wai!:P [19:19:07] thanks Maryana :) [19:25:55] ok, so we have 0 users in oct & nov, and 10 unique non-staff uploaders in december [19:26:01] awjr ^ [19:26:16] woohoo [19:26:27] thanks Maryana [19:26:28] did you want the # of files uploaded in dec, too? [19:26:33] that would be great [19:26:36] no sweat [19:26:46] Maryana: is it likely we actually had more uploads that we weren't able to log? [19:27:04]