[03:35:33] <grrrit-wm>	 (CR) Chmarkine: "Yes. But I am going to change out-links to protocol relative in a different patch, while this one only fixes mixed content vulnerability." [analytics/wikistats] - https://gerrit.wikimedia.org/r/147876 (owner: Chmarkine)
[06:58:42] <grrrit-wm>	 (CR) Gergő Tisza: "In theory, neither domready and onload are really the "right" metric since what we are interested in is when the main image finishes loadi" [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza)
[07:44:20] <grrrit-wm>	 (PS4) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021
[07:45:46] <grrrit-wm>	 (CR) Gergő Tisza: Track loading time for MediaViewer and the file page (1 comment) [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza)
[07:49:40] <grrrit-wm>	 (PS5) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021
[07:53:07] <grrrit-wm>	 (PS2) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148024
[08:17:57] <tgr>	 is there a way to put some custom indexes on EventLogging data?
[08:18:03] <tgr>	 if so, whom should I ask?
[08:59:21] <grrrit-wm>	 (CR) Nuria: "Run code with a 'real' cohort for 'arwiki' and it works well." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) (owner: Milimetric)
[13:08:30] <milimetric>	 hi nuria, if you're around, let's talk about RAE?
[13:28:19] <grrrit-wm>	 (CR) Gilles: Track opt-out ratio (1 comment) [analytics/multimedia] - https://gerrit.wikimedia.org/r/143501 (owner: Gergő Tisza)
[13:31:13] <grrrit-wm>	 (CR) Gilles: [C: 1] "My only remaining concern before +2ing is the lack of index." [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza)
[14:00:53] <kevinator>	 i’ll be in the batcave in a minute… google+ is giving me grief about the right to be there
[14:23:55] <ottomata>	 qchris_meeting: was about to submit another patch for create_webrequest.hql
[14:24:01] <qchris>	 Great.
[14:24:04] <ottomata>	 was thikning about parameterizing the table location
[14:24:09] <ottomata>	 rather than hardcoding it
[14:24:18] <ottomata>	 would be nice if I could set a default, but ja
[14:24:21] <ottomata>	 what do you think?
[14:24:29] <qchris>	 Is that useful to parametrized it?
[14:24:47] <qchris>	 (Currently it's nice that I can just take it and throw it into hive)
[14:24:59] <qchris>	 (without have to set parameters)
[14:25:56] <grrrit-wm>	 (PS1) QChris: Use "adhoc" queue per default [analytics/refinery] - https://gerrit.wikimedia.org/r/148379
[14:25:58] <grrrit-wm>	 (PS1) QChris: Extract HiveQL to create sequence stats table into separate file [analytics/refinery] - https://gerrit.wikimedia.org/r/148380
[14:26:07] <ottomata>	 yeah, i wonder if you can do a default to the param
[14:26:16] <ottomata>	 but, this way you could create a table in your own db without having to edit the query
[14:26:24] <qchris>	 I have not found defaults for hive parameters either.
[14:26:29] <ottomata>	 meh, ok, i won't worry about it then
[14:26:33] <ottomata>	 i was starting to look that up
[14:26:51] <qchris>	 The above change 148380 does the create table statement for the stats table.
[14:26:59] <qchris>	 It's without parameters too.
[14:27:21] <grrrit-wm>	 (PS3) Ottomata: Add webrequest table schema [analytics/refinery] - https://gerrit.wikimedia.org/r/146878
[14:27:44] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Use "adhoc" queue per default [analytics/refinery] - https://gerrit.wikimedia.org/r/148379 (owner: QChris)
[14:28:40] <grrrit-wm>	 (CR) QChris: [C: 2 V: 2] Add webrequest table schema [analytics/refinery] - https://gerrit.wikimedia.org/r/146878 (owner: Ottomata)
[14:28:43] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Extract HiveQL to create sequence stats table into separate file [analytics/refinery] - https://gerrit.wikimedia.org/r/148380 (owner: QChris)
[14:28:46] <ottomata>	 k
[14:39:36] <nuria>	 ottomata, one question ....
[14:39:54] <nuria>	 regarding celery and puppet
[14:41:38] <ottomata>	 yup?
[14:48:47] <nuria>	 ottomata, i will ask you after staff
[14:49:14] <ottomata>	 OH STAFF
[15:36:55] <grrrit-wm>	 (CR) Nuria: [C: 2] Add rolling active editor [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) (owner: Milimetric)
[15:38:11] <grrrit-wm>	 (Merged) jenkins-bot: Add rolling active editor [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/147312 (https://bugzilla.wikimedia.org/67458) (owner: Milimetric)
[15:49:21] <wikibugs>	 Analytics / Wikimetrics: Story:a WikimetricsUser runs 'Rolling Monthly Active Editors' report - https://bugzilla.wikimedia.org/67458 (Dan Andreescu) PATC>RESO/FIX
[15:49:35] <wikibugs>	 Analytics / Visualization: Spike: AnalyticsEng decide on stack for EEVS dashboard - https://bugzilla.wikimedia.org/67172 (Dan Andreescu) NEW>RESO/FIX
[15:50:07] <wikibugs>	 Analytics / Wikimetrics: Optimize JSON format of recurrent report output - https://bugzilla.wikimedia.org/67822 (Dan Andreescu) PATC>RESO/FIX
[15:50:07] <wikibugs>	 Analytics / Wikimetrics: Max recursion limit hit - https://bugzilla.wikimedia.org/67823 (Dan Andreescu) PATC>RESO/FIX
[16:22:10] <grrrit-wm>	 (PS12) Milimetric: Add autcomplete to tags on tag insertion [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/145039 (owner: Terrrydactyl)
[16:22:50] <milimetric>	 nuria, I made the binding generic, take a look and merge if you like it
[17:36:05] <tnegrin>	 technical difficulties getting into hangout for retro
[17:36:26] <Ironholds>	 tnegrin, well, there's your problem.
[17:36:31] <Ironholds>	 They didn't have hangouts in the olden days.
[17:36:35] <Ironholds>	 You can have hangouts or go retro, pick.
[17:36:49] <tnegrin>	 broken locks
[17:36:53] <tnegrin>	 we had those
[18:09:36] <grrrit-wm>	 (PS2) Gergő Tisza: Remove generated files from source control [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148009
[19:34:53] <wikibugs>	 Analytics / EventLogging: Add index on event_action, event_isAnon and event_namespaceId to NavigationTiming tables - https://bugzilla.wikimedia.org/68396 (Tisza Gergő) NEW p:Unprio s:normal a:None I would like to use the NavigationTiming tables to make a dashboard of image loading speeds by...
[19:37:10] <wikibugs>	 Analytics / EventLogging: Add index on event_type to MultimediaViewerDuration tables. - https://bugzilla.wikimedia.org/68397 (Tisza Gergő) NEW p:Unprio s:normal a:None Needed for an image loading time dashboard (see https://gerrit.wikimedia.org/r/#/c/148021 )
[19:39:26] <grrrit-wm>	 (PS6) Gergő Tisza: Track loading time for MediaViewer and the file page [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021
[19:40:25] <grrrit-wm>	 (CR) Gergő Tisza: "Forgot to filter NavTiming on action = view." [analytics/multimedia] - https://gerrit.wikimedia.org/r/148021 (owner: Gergő Tisza)
[19:41:53] <wikibugs>	 Analytics / Wikistats: Add url to stats - https://bugzilla.wikimedia.org/68398 (Alejandro Sánchez Marín) UNCO p:Unprio s:normal a:None Need to add two url to https://wikistats.wmflabs.org/display.php?t=mw  http://enciclopedia.us.es  http://www.ateneodecordoba.org  Thanks in advance
[19:42:14] <grrrit-wm>	 (CR) Nuria: [C: 2] Add autcomplete to tags on tag insertion [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/145039 (owner: Terrrydactyl)
[19:42:25] <grrrit-wm>	 (Merged) jenkins-bot: Add autcomplete to tags on tag insertion [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/145039 (owner: Terrrydactyl)
[19:45:06] <wikibugs>	 Analytics / EventLogging: Add index on event_action, event_isAnon and event_namespaceId to NavigationTiming tables - https://bugzilla.wikimedia.org/68396#c1 (nuria) Adding sean (on ops) to ticket as normally he does the db management of this db. I believe analytics team does not have permits to add ind...
[20:02:20] <wikibugs>	 Analytics / Wikistats: Add enciclopedia.us.es and ateneodecordoba.org URLs to stats - https://bugzilla.wikimedia.org/68398 (Andre Klapper)
[20:07:35] <wikibugs>	 Analytics / Wikistats: Add enciclopedia.us.es and ateneodecordoba.org URLs to stats - https://bugzilla.wikimedia.org/68398#c1 (Oliver Keyes) UNCO>RESO/WON Er. No we don't. Neither of those sites are Wikimedia-run, which means we don't have the request logs for them that would be necessary to actu...
[20:34:13] <grrrit-wm>	 (CR) Gilles: [C: -1] Remove generated files from source control (1 comment) [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148009 (owner: Gergő Tisza)
[21:17:55] <grrrit-wm>	 (PS5) Milimetric: Improve server-side cohort upload form validation [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138151 (owner: AndyRussG)
[21:19:00] <grrrit-wm>	 (CR) Milimetric: "I rebased this and tried to review, but we introduced a bad reference that is causing circular references. I will make a patch and make t" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138151 (owner: AndyRussG)
[21:21:40] * YuviPanda waves at milimetric
[21:22:04] <milimetric>	 hi YuviPanda :)
[21:22:11] <milimetric>	 how you doin
[21:22:49] <YuviPanda>	 milimetric: good :) the dashboard stuff didn't make this sprint, though :(
[21:23:00] <YuviPanda>	 milimetric: also I'll be leaving the mobile apps team soon.
[21:23:08] <milimetric>	 i heard from nuria, yea
[21:23:20] <milimetric>	 well, we did do a prototype, did you see my email YuviPanda?
[21:23:34] <YuviPanda>	 milimetric: yeah, looks nice! :D
[21:23:39] <milimetric>	 it's using mediawiki metadata and rendering vega graphs and everything
[21:23:42] <milimetric>	 consuming wikimetrics data
[21:23:47] <milimetric>	 300 lines of JS!
[21:23:48] <milimetric>	 :)
[21:23:50] <YuviPanda>	 :D nice
[21:23:53] <YuviPanda>	 milimetric: have you seen https://meta.wikimedia.org/wiki/Research:Ideas/Public_query_interface_for_Labs
[21:24:24] <milimetric>	 yea YuviPanda, we talked about it
[21:24:38] <milimetric>	 definitely one way would be for you to submit a patch for an "ad-hoc" metric to wikimetrics
[21:24:38] <YuviPanda>	 milimetric: I've a basic prototype at yuvi.in:5000
[21:24:39] <YuviPanda>	 no queries
[21:26:34] <milimetric>	 that's cool Yuvi
[21:26:57] <milimetric>	 I think wikimetrics would be easier to do, so you don't have to write a whole separate system with users etc.
[21:27:42] <milimetric>	 not to mention async queue
[21:28:29] <grrrit-wm>	 (PS3) Gergő Tisza: Remove generated files from source control [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148009
[21:28:53] <YuviPanda>	 milimetric: right. but the more fundamental idea here is to let people have queries publicly shareable, and 'remixable'.
[21:28:58] <YuviPanda>	 milimetric: like jsbin for sql
[21:29:59] <grrrit-wm>	 (CR) Gergő Tisza: "Ported to python." [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148009 (owner: Gergő Tisza)
[21:30:05] <milimetric>	 right, that'd be a new feature, but why not build it on top of wikimetrics?
[21:30:15] <milimetric>	 you'd get all the stuff that's already solved for free
[21:30:33] <YuviPanda>	 hmm, that's true
[21:30:50] <YuviPanda>	 halfak: ^
[21:31:26] <halfak>	 I was going to suggest the same.
[21:31:43] <halfak>	 :)  When I was looking through the mock of running queries I though of wikimetrics.
[21:31:48] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Remove generated files from source control [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148009 (owner: Gergő Tisza)
[21:31:51] <milimetric>	 the way i see it, we all can win if we make wikimetrics "the tool  to query our public data"
[21:32:54] <YuviPanda>	 hmm, I'll think about it some more
[21:33:20] <milimetric>	 i don't mean undue pressure Yuvi, if you see a reason for something separate, by all means
[21:33:40] <YuviPanda>	 no, I haven't thought about the idea of just adding this to wikimetrics at all :D
[21:33:45] <YuviPanda>	 so now I should
[21:34:00] <milimetric>	 let me know if you want to go over the details of what we'd need and how it might integrate
[21:34:30] <YuviPanda>	 milimetric: right. I'll also admit to not having used wikimetrics more than cursorily, so I should do that first
[21:34:52] <milimetric>	 YuviPanda: it's tailored to think about the world in terms of "cohorts" and "metrics"
[21:35:18] <milimetric>	 we've already expanded "cohorts" to also be able to use entire projects (like enwiki, commonswiki)
[21:35:23] <YuviPanda>	 right, where I presume a 'cohort' is a list of userids  and metrics are timeseries?
[21:35:31] <milimetric>	 yes, initially
[21:35:46] <milimetric>	 metrics are not timeseries by default, but that doesn't matter for your use case
[21:35:55] <milimetric>	 basically, to implement ad-hoc querying, we'd need three things:
[21:36:12] <milimetric>	 an "ad-hoc" metric whose results don't go through the normal individual / timeseries / aggregate pipeline
[21:36:34] <milimetric>	 to open up project-level cohorts to everyone (something we've been a bit afraid to do for perf. reasons, but you're doing it anyway so it doesn't matter
[21:37:00] <milimetric>	 and finally to build the kind of "jsbin" social remixing thing you're talking about
[21:37:27] <milimetric>	 it's not completely clear cut but I think those first two items are easier than bootstrapping and building a new project
[21:37:55] <YuviPanda>	 (2) will need a fair chunk of massaging
[21:38:13] <milimetric>	 making project-level cohorts available?
[21:38:47] <YuviPanda>	 things I've thought of so far 1. Limit total number of queries, 2. kill queries after 5-10min 3. limit number of concurrent queries per user 4. make users 'blockable' (by blocking / revoking shell on wikitech) 5. sql parse the SQL and take that into account
[21:38:49] <milimetric>	 that's the easy item here, they already work great, we'd just have to allow everyone to see them and add them to their list
[21:39:16] <YuviPanda>	 right, I was referring to the performance impact
[21:39:18] <milimetric>	 right, YuviPanda, that's why we use celery
[21:39:33] <milimetric>	 it has soft/hard timeouts for all the tasks
[21:39:37] <milimetric>	 and we can play with those
[21:40:42] <grrrit-wm>	 (PS1) Gergő Tisza: Fix generate.py [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148511
[21:41:04] <grrrit-wm>	 (CR) Gergő Tisza: [C: 2] "Self-merge, trivial fix." [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148511 (owner: Gergő Tisza)
[21:41:29] <YuviPanda>	 milimetric: hmm, right. This OAuth's to mediawiki.org for auth?
[21:41:55] <milimetric>	 yes, people can pick google login or MW
[21:42:03] <milimetric>	 the MW one just goes to meta
[21:42:07] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Fix generate.py [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148511 (owner: Gergő Tisza)
[21:42:12] <milimetric>	 but we can point one to wikitech, no matter
[21:42:58] <YuviPanda>	 milimetric: right, the fundamental reason being you need to agree to labs ToS explicitly before you can use the SQL
[21:44:08] <milimetric>	 hm, it could be that wikimetrics delegates that right, but we should talk to labs folks.  Either way, we could allow only wikitech users to use the ad-hoc feature if we want
[21:46:29] <YuviPanda>	 milimetric: right. I guess wikimetrics doesn't need it because it's not freeform access..
[21:46:58] <milimetric>	 yeah, might be, i am not a lawyer :)
[21:48:33] <YuviPanda>	 milimetric: :D
[21:48:45] <YuviPanda>	 milimetric: I'll try it out and look over it some more tomorrow :)
[21:48:58] <YuviPanda>	 milimetric: having 'one place to run analyticsy stuff for people outside the fold' makes sense
[21:49:11] <YuviPanda>	 but I'm wary of force fitting different concepts into one tool just for that sake
[21:49:26] <YuviPanda>	 that's how we end up with mediawiki running commons and mediawiki running wikitech and so on
[21:49:27] <milimetric>	 well, I agree completely, just see if it makes sense
[21:49:32] <YuviPanda>	 yeah, totally
[21:50:00] <milimetric>	 we've talked about adding an ad-hoc metric and there was some opposition and some support on our side too
[21:50:06] <milimetric>	 i've always been for it
[21:50:41] <YuviPanda>	 what was the opposition's points?
[21:51:04] <grrrit-wm>	 (PS1) Gergő Tisza: Add missing date limit to one of the action UNION ALLs [analytics/multimedia] - https://gerrit.wikimedia.org/r/148518
[21:51:45] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Add missing date limit to one of the action UNION ALLs [analytics/multimedia] - https://gerrit.wikimedia.org/r/148518 (owner: Gergő Tisza)
[22:01:02] <tgr>	 milimetric: what would be the nice way of supporting a custom build script in limn-deploy?
[22:01:29] <tgr>	 that is, after checking out the data repository, run a custom script to generate the json files from templates
[22:01:50] <tgr>	 I have added a git hook for now, but that is a bit awkward
[22:02:38] <tgr>	 so I was thinking of writing a patch for limn-deploy, just wondering about the right approach
[22:02:54] * milimetric is thinking about tgr's question
[22:02:58] <tgr>	 have a hook directory in the repo and if the file exists, automatically execute it?
[22:03:18] <tgr>	 in the data repo I mean
[22:04:10] <milimetric>	 tgr: I think maybe having a setting that's stage-specific that is like "post-data-checkout-script" or something like that, one sec... lemme revisit that code
[22:05:42] <milimetric>	 yeah, like adding a setting here:
[22:05:42] <milimetric>	 https://github.com/wikimedia/limn-deploy/blob/master/fabfile/stages.py#L310
[22:05:42] <milimetric>	 and then using it here:
[22:05:42] <milimetric>	 https://github.com/wikimedia/limn-deploy/blob/master/fabfile/deploy.py#L181
[22:06:02] <milimetric>	 iff env.that_setting is set
[22:06:12] <milimetric>	 just run the script it points to, would that work tgr?
[22:06:25] <tgr>	 ah
[22:06:39] <tgr>	 didn't realize all that's hardcoded
[22:06:59] <milimetric>	 the checkout / update stuff or the settings?
[22:07:03] <tgr>	 in that case, yes, I can just add the setting to the fabric config
[22:07:08] <tgr>	 the settings
[22:07:31] <milimetric>	 ah, yep, they were too different everywhere to have any convention when I picked up this repo
[22:07:43] <tgr>	 I'll do that, thanks
[22:07:51] <milimetric>	 np
[22:07:57] <YuviPanda>	 milimetric: when you've time, you should tell me about what the people who apposed adhoc queries said :)
[22:10:01] <milimetric>	 YuviPanda: I think one of the main ones was that wikimetrics aims to provide "standard" queries that people should run so researchers can compare more apples to apples rather than creating their own metrics
[22:10:12] <YuviPanda>	 right
[22:10:31] <milimetric>	 i do think the same argument can be made against any ad-hoc sql solution though
[22:11:33] <milimetric>	 tnegrin / kevinator: you can weigh in if you'd like
[22:11:42] <milimetric>	 YuviPanda is building an ad-hoc sql query tool
[22:11:57] <YuviPanda>	 (https://meta.wikimedia.org/wiki/Research:Ideas/Public_query_interface_for_Labs)
[22:12:05] <milimetric>	 so given that such a thing will exist when he's done, I suggested he might as well stick it into wikimetrics
[22:12:30] <milimetric>	 he's currently thinking it over and if you have counter-arguments it would be useful
[22:12:44] <kevinator>	 interresting…
[22:14:57] <YuviPanda>	 my original (and current) plan is to just build a separate tool, along the lines of jsbin, but for SQL + backed by labsdb
[22:15:35] <tnegrin>	 why can't you help us with the page view api?
[22:15:59] <YuviPanda>	 hmm? these are completely different, no?
[22:16:30] <tnegrin>	 sure -- but people have been asking for page views for years
[22:16:49] <tnegrin>	 we'd love to ship some data into labsdb
[22:16:53] <wikibugs>	 Analytics / Wikimetrics: forms module is poorly nested inside controllers - https://bugzilla.wikimedia.org/68410 (Dan Andreescu) NEW p:Unprio s:normal a:None The wikimetrics/controllers/forms folder causes anyone who imports from forms to import the entire controller suite.  Also, the wikim...
[22:17:22] <YuviPanda>	 indeed, and people have been trying to do something like this for a while too. Toolserver had a wiki page where you can request queries be run for you, I think...
[22:17:22] <wikibugs>	 Analytics / Wikimetrics: forms module is poorly nested inside controllers - https://bugzilla.wikimedia.org/68410 (Dan Andreescu)
[22:17:41] <YuviPanda>	 and the researchers I talked to think this would definitely gain a lot more exposure to labsdb as well...
[22:17:45] <tnegrin>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=42259
[22:18:53] <milimetric>	 :) while I hesitate to interrupt toby from trying to convince Yuvi to work with us (Totally do it Yuvi, we <3 you), I think we can talk about the wikimetrics question separately
[22:19:04] <tnegrin>	 ;)
[22:19:13] <Ironholds>	 YuviPanda, do iiiiit.
[22:19:22] <tnegrin>	 I love yuvi too but I want my current projects to go faster
[22:19:23] <Ironholds>	 do it and I'll buy you a 30 year old single malt.
[22:19:28] <YuviPanda>	 hehe :D
[22:19:38] <YuviPanda>	 Ironholds: what, the pageview stuff or the query thing?
[22:19:52] <Ironholds>	 how the hell should I know, I'm not a product manager
[22:19:55] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Track opt-out ratio [analytics/multimedia] - https://gerrit.wikimedia.org/r/143501 (owner: Gergő Tisza)
[22:19:56] <YuviPanda>	 :D
[22:19:56] <YuviPanda>	 hehe
[22:19:57] <Ironholds>	 I just want more yuvi in our analytics stuff
[22:20:05] <tnegrin>	 for sure
[22:20:14] <YuviPanda>	 tnegrin: heh :D
[22:20:26] <YuviPanda>	 me too, esp. now I won't have to do ops stuff 'on the side' soon.
[22:20:38] <tnegrin>	 yeah -- congrats
[22:20:47] <YuviPanda>	 :) ty
[22:21:13] <YuviPanda>	 tnegrin: other than the 'I want my projects to go faster!' I guess you don't have anything against me doing the query thing (inside wikimetrics or not)? :)
[22:21:20] <tnegrin>	 of course not
[22:21:28] <kevinator>	 YuviPanda: how long is the sql interface project anyway?
[22:21:38] <tnegrin>	 but you asked ;)
[22:21:47] <YuviPanda>	 kevinator: my personal estimate is 2 months and then maintenance
[22:22:26] <YuviPanda>	 tnegrin: :D I'll be happy to help on that stuff after. I do think more analytics stuff needs to come into labs
[22:22:28] <tnegrin>	 if dan/kevin think it belongs in WM, I agree with them
[22:22:36] <tnegrin>	 yeah….we need to free the data!
[22:22:54] <YuviPanda>	 yeah, lots of people are using the dumps for things that'll be far easier with labsdb
[22:22:57] <tnegrin>	 we need to integrate EL, Hadoop, the databases and labs
[22:23:06] <tnegrin>	 this is true
[22:23:07] <YuviPanda>	 the question of it belonging in WM is both a product and technical one, I guess.
[22:23:35] <YuviPanda>	 indeed, hadoop on labs with something as trivial as wikidata or enwiki dump would be awesome to start with
[22:23:45] <tnegrin>	 will your tool support scheduled queries?
[22:23:55] <kevinator>	 milimetric: where in WM would it go?  under “create report” I would get a tab to enter my own SQL?
[22:23:59] <tnegrin>	 you get some scaffolding in WM you might not want to do yourself
[22:24:08] <YuviPanda>	 tnegrin: cronish? no, no plans for that atm.
[22:24:16] <tnegrin>	 yeah
[22:24:25] <grrrit-wm>	 (PS3) Gergő Tisza: Track opt-out ratio [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148006
[22:24:34] <YuviPanda>	 but yeah, that could be potentially added. the idea being 'play around with this thing, but if you want to do hardcore stuff get a toollabs account'
[22:24:35] <milimetric>	 kevinator: it could be another metric
[22:24:41] <milimetric>	 "Ad-hoc SQL"
[22:24:56] <tnegrin>	 also python notebook is another possible integration point
[22:24:59] <YuviPanda>	 a gateway drug of sorts
[22:25:10] <tnegrin>	 ipython that is
[22:25:17] <milimetric>	 so you could pick whatever cohort, and parametrize it with templated sql maybe, and then you could schedule it recurrent or whatever wikimetrics offers
[22:25:18] <YuviPanda>	 tnegrin: indeed, I'm closely following ipython notebooks' multiuser support. Once that gets merged I'll have a labs setup started after :)
[22:25:25] <tnegrin>	 sweeet
[22:25:43] <YuviPanda>	 tnegrin: we can't really deploy it before that because security :)
[22:25:52] <tnegrin>	 long term vision is that we can present some unified schema and a collab env
[22:26:09] <tnegrin>	 damn security
[22:27:02] <kevinator>	 what about a requirement to prevent users from dropping tables or writting to them?
[22:27:14] <YuviPanda>	 milimetric: right, so templated SQL wasn't something I was planning on supporting either. I was, however, planning on adding code-complete type stuff for the SQL (derived from tables.sql), indexes, etc
[22:27:24] <YuviPanda>	 kevinator: that's already handled for us at labsdb. don't have the rights ;)
[22:27:38] <kevinator>	 got it
[22:27:48] <kevinator>	 I gotta run to a meeting...
[22:28:07] <YuviPanda>	 the focus of the tool I was planning on would be on exploratory SQL + sharing/learning-from-others
[22:28:14] <YuviPanda>	 more JSBin than anything else
[22:28:40] <milimetric>	 yeah, I mean, one thing you could do is keep building out the prototype and if you start running into the problems that wikimetrics solves, reconsider
[22:28:46] <lzia>	 hey nuria, are you around?
[22:29:36] <milimetric>	 those would be: async queueing, scheduled runs, some light user management, public results hosting, etc.
[22:29:47] <milimetric>	 lzia: she said she was done a couple hours ago I think
[22:29:55] <lzia>	 ah! thanks!
[22:30:31] <lzia>	 Do you know if/how I can re-install R on stat1003, by any chance, milimetric?
[22:30:54] <YuviPanda>	 milimetric: right, will do! other than (1), none of the others were planned for this, but public results hosting might end up happening.
[22:31:01] <milimetric>	 lzia: I think only andrew has rights to do that, or any ops folks
[22:31:15] <lzia>	 aha! thanks! that's helpful.
[22:31:24] <milimetric>	 YuviPanda: do you have root on stat1003?
[22:31:37] <YuviPanda>	 milimetric: heh, remember, puppet :)
[22:32:20] * milimetric stares blankly into the void and mumbles... "puppet, what is puppet?"
[22:32:30] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Track opt-out ratio [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148006 (owner: Gergő Tisza)
[22:32:30] <YuviPanda>	 milimetric: lzia I'm submitting a puppet change to install r on stat 1003
[22:32:55] <lzia>	 it needs another layer of translation YuviPanda. ;-)
[22:33:18] <lzia>	 I'm blocked by it now, aren't you? :-)
[22:35:03] <milimetric>	 thanks Yuvi, you rock, I don't get how it got removed, that would be interesting if you find out
[22:38:02] <YuviPanda>	 lzia: milimetric uh, it *should* be installed?
[22:38:09] <YuviPanda>	 I just followed the maze of includes, and I see r-base there
[22:38:37] <YuviPanda>	 lzia: are you sure it isn't there?
[22:38:54] <lzia>	 humm. I am seeing 2.14.1
[22:39:01] <lzia>	 YuviPanda, ^
[22:39:45] <YuviPanda>	 lVersion: 2.14.1-1
[22:39:49] <YuviPanda>	 lzia: right, so that's the version on precise
[22:40:09] <YuviPanda>	 lzia: if you need a newer version, then ugh, either a new machine with trusty needs to be installed or someone has to 'backport' the package and get it accepted into apt.wikimedia.org
[22:40:14] <YuviPanda>	 so, uh, in short, not fun
[22:40:47] <YuviPanda>	 if we upgrade to trusty we get 3.0.2
[22:41:04] <YuviPanda>	 milimetric: lzia ^
[22:41:05] <lzia>	 isn't 3.1.1 out?
[22:41:13] <lzia>	 YuviPanda, ^
[22:41:22] <YuviPanda>	 lzia: probably, and packages in ubuntu usually lag behind actual releases
[22:41:29] <YuviPanda>	 http://packages.ubuntu.com/trusty/math/r-base would be the trusty version
[22:41:34] <lzia>	 I want to install bunch of libraries and they are confuzzled with the old version.
[22:41:45] <lzia>	 lemme check that, YuviPanda
[22:43:07] <wikibugs>	 Analytics / EventLogging: Add index on event_action, event_isAnon and event_namespaceId to NavigationTiming tables - https://bugzilla.wikimedia.org/68396 (Sean Pringle) a:Sean Pringle
[22:43:44] <YuviPanda>	 can't compile it yourself either, the machine doesn't have appropriate build tools :(
[22:44:08] <lzia>	 yup, YuviPanda. Thanks for the help. Let me dig in more if I can find a work around.
[22:44:16] <lzia>	 tnx milimetric
[22:45:02] <milimetric>	 sorry lzia, if you're completely stuck, raise it up and tell us what you're trying to do, maybe there's another way in R / outside R
[22:45:11] <YuviPanda>	 lzia: :) it's possible to compile R yourself locally and just use it. That way you can always have the latest version. it just needs some tools that we can install now itself, and is probably the best short workaround.
[22:45:39] <YuviPanda>	 am trying out that possibility on toollabs now, to see how hard it is
[22:45:40] <wikibugs>	 Analytics / EventLogging: Add index on event_type to MultimediaViewerDuration tables. - https://bugzilla.wikimedia.org/68397 (Sean Pringle) a:Sean Pringle
[22:47:43] <lzia>	 yeah, the problem is that right now I am running huge sets of data and local doesn't work any more. :-(
[22:48:00] <YuviPanda>	 lzia: no, not locally on your machine I meant. in your home folder on stat1003
[22:48:01] <lzia>	 I tried it but it resulted in the death of local R
[22:48:11] <lzia>	 ow that!
[22:48:34] <lzia>	 good point!
[22:49:31] <YuviPanda>	 I'm doing it manually on toollabs to see what all needs to be done
[22:52:41] <grrrit-wm>	 (PS1) Milimetric: Fix bad dependency chains and imports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/148537 (https://bugzilla.wikimedia.org/68410)
[22:52:57] <grrrit-wm>	 (PS6) Milimetric: Improve server-side cohort upload form validation [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138151 (owner: AndyRussG)
[22:57:11] <grrrit-wm>	 (CR) Milimetric: "I had to make your patch depend on a long-needed cleanup of the wtforms custom forms and fields that we have floating all over the codebas" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138151 (owner: AndyRussG)
[23:04:51] <YuviPanda>	 bah
[23:04:57] <YuviPanda>	 both lzia and milimetric left
[23:06:43] <grrrit-wm>	 (PS1) Gergő Tisza: Fix JSON breakage [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148538
[23:07:41] <grrrit-wm>	 (CR) Gergő Tisza: [C: 2] "Self-merge, trivial fix." [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148538 (owner: Gergő Tisza)
[23:21:31] <grrrit-wm>	 (CR) Gergő Tisza: [C: 2] Fix JSON breakage [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148538 (owner: Gergő Tisza)
[23:21:55] <grrrit-wm>	 (CR) Gergő Tisza: [V: 2] Fix JSON breakage [analytics/multimedia/config] - https://gerrit.wikimedia.org/r/148538 (owner: Gergő Tisza)