[00:05:07] this can't be [00:05:09] is gerrit down? [00:06:33] gerrit seems to be fine with me [17:21:48] morning everyone [17:22:40] mornin [17:29:55] hey milimetric. [17:29:57] meeting today as well? [17:30:04] hey Yuvi [17:30:14] I have a few meetings, but nothing crazy [17:30:26] the last two days were 100% meetings :) [17:30:37] milimetric: I've the csv data required for all our reports. But can't get limn to run even the example datasets. [17:30:48] I've a meeting with DarTar right now, I guess I can poke you afterwards. [17:30:57] you're trying it locally? [17:31:33] also, commit the datasources and graphs you're trying to make here: https://github.com/wikimedia/limn-mobile-data/ [17:31:54] and if you don't know what I'm talking about :) we should have a quick google hangout so we can sort it out [17:32:34] milimetric: yes [17:32:54] milimetric: I am trying to get the datasources and graphs from limn-data working [17:32:58] milimetric: have you mentioned limnpy? [17:32:58] (suggested on the README) [17:33:01] i get a blank one [17:33:37] erosen: not yet, but maybe it'd be a good idea here [17:35:00] YuviPanda: https://github.com/wikimedia/limnpy [17:35:16] * YuviPanda clicks [17:35:19] I think having a hangout and going through this together might be useful though [17:35:49] I'm curious to see what problems you're having so I can improve the process [17:35:57] milimetric: sure! [17:36:11] milimetric: I'm still waiting on DarTar - he sent an email saying he'd be a little late [17:36:22] hm, let's do it now then [17:36:26] i'll make a hangout [17:36:43] https://plus.google.com/hangouts/_/eba82faac9e77fafe93e0189cac396562e79fbc0 [17:36:53] sure [17:36:54] * YuviPanda clicks [17:37:19] milimetric: i'm in another hangout right now, but let me know if you need any comments from me [17:37:21] no prob [17:37:33] hey YuviPanda [17:37:36] i'd be more than happy to help, though [17:37:37] hey DarTar [17:37:40] milimetric: i can't hear you [17:37:43] just made it to the office [17:37:47] hang out? [17:37:58] I just got on one with milimetric [17:37:58] let me add one to the cal invite [17:38:06] milimetric: can I do it with DarTar and come back to you right after? [17:38:08] ah ok, can I crash it :) [17:38:15] ah that too [17:38:18] DarTar: just drop by! [17:38:23] i forgot you two are in the same physical location [17:38:24] link? [17:38:29] oh right [17:38:34] milimetric: no audio yet [17:38:37] oh I'm just at my desk [17:38:37] milimetric: can I join for a few mins? [17:38:37] https://plus.google.com/hangouts/_/eba82faac9e77fafe93e0189cac396562e79fbc0 [17:38:44] of course DarTar ^^ [17:38:46] I have no audio [17:38:51] oh [17:38:53] I booked R63 on 6 [17:38:58] so I'll drop and come back, linux - sheesh [17:38:58] if you prefer a room [17:39:05] haha [17:42:36] YuviPanda, DarTar I'm coming up because it's super noisy here [17:42:49] oh nvm - my standup is soon [17:59:23] milimetric: i'm skipping scrum today; GLEE sprint planning got rescheduled for now [17:59:35] ooh, you have something called GLEE? [17:59:37] that's awesome [18:00:33] milimetric: my standup is now, I'll be back in 10 [18:00:45] me too YuviPanda, I'll leave the hangout open [18:01:00] milimetric: Grantmaking Learning and Evaluation (something with an E) [18:04:31] drdee: until when are you in town? [18:05:49] I want to run past you guys some ideas around representing funnels as directed edge lists from EL data (resurrecting the early convo we had a few weeks ago) [18:09:07] hey milimetric. ping me when your standup's done [18:36:37] YuviPanda: back [18:36:51] wanna hop back into the hangout? [18:36:59] milimetric1: 5 mins. [18:37:00] milimetric: [18:37:26] cool [18:42:26] morning [18:42:30] average_drifter: around? [18:43:59] milimetric: coming up [18:44:09] drdee: yes, I'm here, I'm looking at the bug [18:44:39] ok [18:51:04] [travis-ci] master/1fc2359 (#69 by Diederik van Liere): The build has errored. http://travis-ci.org/wikimedia/kraken/builds/5291630 [19:04:18] [travis-ci] master/b3e5246 (#70 by Diederik van Liere): The build has errored. http://travis-ci.org/wikimedia/kraken/builds/5292088 [19:36:20] drdee: http://mobile-reportcard.wmflabs.org/graphs/pageviews_mobile_by_manufacturer [20:27:50] question [20:28:16] ottomata: what input did you try the filter on when it SEGFAULTed ? [20:28:42] it was pre-space_to_tab [20:28:50] now the code splits on tab [20:28:53] that's why it's segfaulting :) [20:29:48] uhm [20:34:00] average_1rifter [20:34:04] i tried it only on the live stream [20:37:50] milimetric: i still don't have perms to that repo [20:37:53] ottomata: live stream has tabs? [20:38:02] yes [20:38:08] oops YuviPanda, adding [20:38:30] ottomata: can you please help me and multiplex live stream and feed it once to the filter and once to a file on disk [20:38:44] hmm, lemme just grab some of the logs and we can try it [20:38:44] ottomata: when it segfaults I need the file written on disk to see what happens [20:38:48] ok [20:39:21] YuviPanda, that's right, I don't have the privilege to add you - you have to ask someone who does (don't know who) [20:39:28] milimetric: asking brion [20:42:55] average_1rifter, i did a short tail of sampled logs and didn't segfault [20:43:00] gonna try again in prod just to be sure [20:43:13] ok [20:44:42] yeah still segfaulting [20:44:43] hm [20:45:35] ok, if you have a small sample, please let me know where I can get it [20:45:40] from the livestream I mean [20:45:44] the one that made it segfault [20:45:47] i will try to get you one [20:45:52] thank you [20:46:11] milimetric: pushed [20:50:12] average_1rifter, what package provides db.h again? [20:51:15] libdb [20:51:40] libdb4.8 [20:51:52] ottomata: ^^ [20:52:08] locke is running libdb4.6, is that related? [20:55:08] ok, it seg faults on stat1 too [20:55:11] average_1rifter [20:55:44] hmm [20:55:51] don't think libdb has to do with it [20:55:52] /a/otto/webstatscollector/filter < /a/otto/unsampled.tab.log [20:56:02] you can reproduce with that log [20:56:05] on stat1 [20:56:06] ok so the segfaulting input is there [20:56:08] I'm scp-ing it now [20:56:09] ja [20:56:11] k [20:56:46] ottomata: thank you [20:57:31] yup [20:58:15] milimetric: ping me when you're around :) [21:09:22] milimetric: do you have an ETA on when you might be available to help set those CSVs up on limn? [21:09:27] (asking to help plan my sleep :P) [21:11:35] YuviPanda: looks like dan is in a meeting until 2 [21:11:57] YuviPanda: hi, btw [21:12:07] hey erosen :) [21:12:23] hmm, so that's two more hours. Damn I missed his free time. [21:12:31] only 1 [21:12:39] oh right [21:12:49] what step are you stuck on? [21:12:52] my pst calculator is broken in my head [21:12:59] erosen: so I can't seem to setup limn on my local machine [21:13:03] YuviPanda: it's possible I can help [21:13:05] it just doesn't work. [21:13:06] aah [21:13:06] erosen: sure! [21:13:10] erosen: so we both did a hangout [21:13:12] and screen shared [21:13:17] and checked out why it didn [21:13:22] and couldn't find anything wrong [21:13:28] but I got nothing on local machine [21:13:32] weird [21:13:40] erosen: but he asked me to put up the csvs on https://github.com/wikimedia/limn-mobile-data [21:13:42] and I have [21:13:49] and told me he'll help setup an instance on kripke on labs [21:13:55] erosen: you can help do that? :D [21:14:04] i can't help with the limn install, but I might have some ideas about the csvs [21:14:21] sure! the script used to generate them is also in the repo [21:14:40] it runs on stat1, but is not a cronjob or anything yet [21:15:08] gotcha [21:15:41] i sort of know how the kripke set up works, but I'm wary of giving out of date info, as the deploy process has been changing pretty quickly lately [21:17:39] erosen: hmm [21:17:45] erosen: I can ssh into it, but that's pretty much it [21:17:57] YuviPanda: I'm just checking things out now [21:17:57] I dunno where the datafiles or, or where I can add graph / datasource configs [21:18:04] okay! [21:18:30] the one I'm most familiar with is /srv/gp.wmflabs.org/ [21:19:18] I could probably get through the setup process manually, but i know that dan has a deploy script to all of this [21:19:40] erosen: hmm, yeah. if it is just 40 mins more I could definitely wait :) [21:19:56] seems like the best option [21:20:03] yup! :) [21:20:17] erosen: in the mean time, if you do have any opinions on the script I'm using to generate the data, do tell! [21:20:17] I'm famiilar with the datasource / graph file layout, but can't help much on the deploy end [21:20:30] (https://github.com/wikimedia/limn-mobile-data/blob/master/mobile-app-stats.py) [21:20:38] the csv generation seemed pretty good [21:20:52] i've done a lot of automated generation of the associated datasources. graphs and dashboards [21:21:05] which is where limnpy (github.com/wikimedia/limnpy) comes in [21:21:17] ah right [21:21:23] but I think dan has been working on a way to infer the datasource (metadata) from the raw csvs, [21:21:35] so it's not clear whether that step is necessary [21:21:41] hmmm [21:21:47] but even then, it should be relatively little work, I suppose? [21:21:58] these are straight-simple line graphs [21:22:00] do yo mean to make the datasource? [21:22:12] or the graphs [21:24:26] erosen: both? [21:24:32] I mean, the limn config files. [21:24:37] yeah, those are pretty simply [21:24:40] simple* [21:25:01] I have a little script which does this in python, if you think that the config files are the missing piece [21:25:10] erosen: sure! Link? [21:25:17] erosen: but I need somewhere to test them, since I don't have a working local instance [21:25:28] it's a console script associated with limnpy [21:25:40] https://github.com/wikimedia/limnpy#command-line-utility [21:26:21] but form the looks of mobile-reportcard.wmflabs.org/ it seems like you need a limn install as well [21:26:43] erosen: yup. but let me play around with limnpy [21:26:47] erosen: do you have a local limn install? [21:26:52] yeah [21:26:55] i haven't used it in a bit [21:26:57] but I do have one [21:27:31] i'm checking now [21:27:51] ok [21:28:01] whether it works, that is [21:28:23] server mostly works [21:28:27] i'll try adding some data files [21:28:30] okay! [21:30:21] YuviPanda: seems to be working [21:30:49] erosen: can you help setup datasources for the csvs in that repo? [21:30:50] i'm using 80af46ea70ef5c2c4e2b578983f2dabbcf59cc80 on master [21:31:01] sure [21:31:59] erosen: sweeet! [21:33:30] oo [21:33:36] i may have found the problem [21:33:43] erosen: ooh? [21:33:52] i think limn usually expects dates to be formatted with backslash rather than dash [21:34:21] though this may just be my own script being unnecessarliy paroachial [21:34:51] erosen: my limn issues were that I couldn't get limn to work even with the sample dataset [21:36:04] k, working on creating datasources, one sec [21:40:09] YuviPanda: success [21:40:25] the whole date separator thing was a false trail [21:40:44] erosen: sweet! Are you committing to the repo? [21:40:53] sure [21:41:00] i was just starting to wonder how to get you the code [21:41:01] one sec [21:41:05] :D [21:43:44] YuviPanda (testing on stat1 so that your data generation script just creates the datasources and graphs) [21:46:59] erosen: okay [21:47:34] so guys, we had standup earlier, but we didn't figure out what to do about the mobile deliverables [21:47:41] especially the data crunching bit [21:47:55] dschoon_, you mentioned wanting to help work on that [21:48:04] is that something we could actually work on together? [21:51:46] erosen: you have push access to that repo, right? [21:51:55] i think so [21:53:16] okay! [21:54:45] YuviPanda: pushed [21:54:55] you should just need to install limnpy [21:55:26] and maybe change your mysql auth line at the beginning of the script (sorry i had to change it so it worked with my setup) [21:55:32] heh okay [21:55:33] checking [22:00:24] YuviPanda: any luck? [22:00:52] erosen: weird ssh bug not letting me log in. [22:00:57] to kripke? [22:01:11] to stat1. [22:01:13] or anywhere, really [22:01:18] blocks at trying to send LC_ALL [22:01:19] hmm [22:01:20] debugging it [22:01:33] k, well let me know when you get past ssh [22:01:42] sure! [22:11:41] erosen: fixed that, now setting up limnpy [22:15:43] ottomata: yeah, let's talk [22:15:57] i have one more thing to finish for the metrics meeting tomrorow [22:15:59] which should be brief. [22:19:59] YuviPanda: great! how is it going? [22:28:11] File "/home/yuvipanda/mobile-uploads/src/limnpy/limnpy/__init__.py", line 1 [22:28:12] """ [22:28:12] erosen: [22:28:13] ^ [22:28:31] hmm [22:28:47] did you run pip install on it? [22:28:54] wait [22:28:59] let me get you full stacktrace [22:29:06] cool [22:29:53] erosen: http://pastebin.com/V2b0Uj8Q [22:29:59] that's probably more useful :) [22:30:14] indeed [22:30:32] let me check if I somehow am running a different version [22:30:37] it's in a virtualenv [22:30:44] but that's a weird error [22:31:48] erosen: installed with pip install -e git+git://github.com/wikimedia/limnpy.git#egg=limnpy-0.1.0 [22:31:54] is there a newer version? [22:32:03] should be up to date [22:32:08] I just pushed a few small changes [22:32:16] wait [22:33:30] hmm [22:33:30] ok [22:33:38] something weird is going on with my repo [22:33:44] it was just migrated to wikimedia [22:34:47] aah, I was on a weird branch [22:34:48] one sec [22:34:48] ah [22:35:28] ok [22:35:32] give that install a try now [22:36:08] (btw, the version number in the install command doesn't actually matter, it is just used by pip i believe for internal bookkeeping / dependency tracking) [22:36:17] erosen: heh [22:36:17] ok [22:36:20] erosen: doing [22:37:01] erosen: whippee that works :D [22:37:05] yay [22:37:14] just the install I presume [22:37:18] or the whole script? [22:37:33] erosen: no, the script [22:37:35] I see a data/ [22:37:36] niice [22:37:40] great [22:37:52] :) [22:37:56] so those data-sources/files and graphs worked on my local limn [22:38:19] which was running the limn master HEAD [22:39:03] erosen: nice. [22:39:08] erosen: so last step is to deploy :) [22:39:12] * YuviPanda pokes milimetric [22:40:51] hey YuviPanda [22:41:00] hey milimetric [22:41:04] what's up? [22:41:07] oh deploying [22:41:10] yeah [22:41:27] dschoon and I are going to do that [22:41:28] right now [22:41:31] okay [22:41:33] no more delays :) [22:41:35] whee [22:41:45] were you able to get the limn stuff set up? [22:41:50] milimetric: locally? no [22:41:54] but we have data sources, etc done [22:41:57] let me commit [22:41:59] one second [22:42:28] cool [22:42:45] yeah, we'll have to look at your local setup soon to figure out why it's not working [22:43:19] yeah [22:44:41] milimetric: pushed [22:44:44] cool, thx [22:44:46] pulling [22:45:14] milimetric: I like keeping the data on github. [22:45:22] Cool, that works [22:45:41] milimetric: IIRC labs and stat1 have no access to each other, so me and ori are setting up a simple process that constantly pushes data from stat1 to repos [22:45:57] milimetric: not yet pushed. network error [22:45:59] eugh [22:45:59] pushing [22:46:19] milimetric: pushed [22:47:04] ok, gonna have a patch [22:47:20] ottomata: can you please help me in testing this on the live stream ? [22:47:38] YuviPanda what name would you like? something.wmflabs.org? [22:47:51] milimetric: we already pointed mobile-reportcard.wmflabs.org no? [22:47:53] to kripke? [22:48:12] yeah, just wondering if you wanted that or something else [22:48:20] nah that's cool [22:48:40] so we'll make this one right now mobile-reportcard-dev so we can have separate "dev" and "stable" instances [22:49:06] milimetric: okay [22:49:19] but would that be needed? all it is doing is running limn [22:49:23] even the dat ageneration is on stat1 [22:50:12] yeah, we want to keep Tomasz's stable stuff on mobile-reportcard and new stuff that we're working on mobile-reportcard-dev [22:50:22] ooh, okay [22:54:24] will take a few minutes [22:54:34] DarTar: what name do you want for your dashboards? [22:54:57] like a slug? [22:54:58] or title [22:55:00] ? [22:55:13] or labs domain? [22:59:34] yeah, labs domain [23:00:15] ^^ DarTar [23:00:31] hmm, ee is taken right? [23:00:53] these are dashboards for both e3 and e2 stuff [23:01:01] milimetric: how are we going to do data updation? [23:01:15] milimetric: a far simpler method seems to just put the data out on stat1's web server [23:01:18] DarTar: ee-dashboard ? [23:01:19] yes [23:01:20] and read it from the labs instance [23:01:22] ^^ DarTar [23:01:24] milimetric: on a cron [23:01:25] or something [23:01:26] +1 [23:01:28] kk [23:01:36] yeah, Panda|DunceHat [23:01:45] that's what kraken would normally do [23:01:45] milimetric:I spoke to Panda|DunceHat earlier that sounds to be like the best option [23:01:55] (interim solution at least ;) ) [23:02:00] yep, definitely [23:02:14] stat1 is fine. the only changes are the URLs in the datasources [23:02:19] so I'm updating the limn-mobile-data datasource files right now Yuvi, they're not in the right format [23:02:20] yep [23:02:32] (not your fault, our fault) [23:03:00] so let's just create a publicly accessible dump folder on stat1, make it writable for a bunch of users who run these scripts and have limn pull data from there, right? [23:03:05] milimetric: my fault probably [23:03:08] yes, agreed Yuvi [23:03:12] I used the same script i use [23:03:17] no erosen, it's probably my fault [23:03:25] this is just too much in flux right now [23:03:36] i'll fix it, no worries [23:04:09] Yuvi, can you please move the datafiles to the stat1 folder and update the url in the datasources accordingly? I'll do my updates and when merged we should be all set [23:04:17] Panda|DunceHat ^^ [23:04:47] milimetric: 'stat1 folder'? which one? [23:05:00] milimetric: I just found out that I don't have write perms to anything under /var/www [23:05:32] DarTar was saying we needed to make a public folder on stat1 so people can put limn-discoverable datafiles there [23:05:40] :) thanks YuviPanda, that's much easier to type [23:05:45] yeah :) [23:05:52] milimetric: you should be using tab autocomplete :P [23:06:02] heh, my crappy IRC client doesn't have [23:06:07] OMG! it does [23:06:11] everyone does :P [23:06:21] * milimetric 's life changed [23:06:51] DarTar: are you going to get ops to create a folder in /var/www that is writeable by a bunch of people? [23:07:04] DarTar: Leslie told me to create an RT ticket [23:07:29] YuviPanda: I can't right now, in the middle of 15 different things, do you wanna do it? [23:07:37] DarTar: sure. let me do that [23:07:41] thx [23:07:46] what do you wanna call it? [23:07:52] el-data [23:07:53] ? [23:07:54] hm [23:08:02] oh I thought something more generic [23:08:14] like, data :) [23:08:15] like? [23:08:17] or csv [23:08:19]