[00:19:33] user@garage:/tmp/r31-mimetype-samples$ cat * | perl -ne 'BEGIN{$wiki_basic = 0; $wiki_index = 0; $api = 0;}; @f=split(/\ /); $url=$f[8]; if($url=~m|\.org/wiki/|){$wiki_basic++} elsif($url=~m|/w/index.php|){$wiki_index++;} elsif($url=~m|/w/api.php|){$api++;}; END{ print "/wiki/ => $wiki_basic\n"; print "/w/index.php => $wiki_index\n"; print "/w/api.php => $api\n"; };' [00:19:39] /wiki/ => 610377 [00:19:42] /w/index.php => 14719 [00:19:44] /w/api.php => 526 [00:19:59] so in 14-31dec 2012 /wiki/ is on top [00:20:17] ip classification follows [00:20:57] drdee: did you ever get the instructions for setting up a local limn install? [00:21:32] i think milimetric put it in the README on the repo [00:21:38] k [00:21:51] just hit an error with the install instructions there, that's why i ask [00:33:25] hey erosen, if you're using the development branch, we just pushed the new limn that can server multiple instances [00:33:42] so the instructions haven't been updated for that [00:33:50] gotcha no worries [00:33:59] i was just trying to test the new graph format and labs was being slow [00:34:06] as for json or yaml, no preference [00:34:11] but I was finally able to get to kirpke [00:34:15] cool [00:34:15] I can't get into labs anymore at all [00:34:18] i switched to json [00:34:25] ok, cool [00:34:40] here is a successful graph with the new format: http://gp-dev.wmflabs.org/graphs/new_limn_test [00:34:46] milimetric: ^ [00:34:56] i'll update install instructions tomorrow [00:35:03] ah, very cool [00:35:23] milimetric: is the default color for one line black? [00:35:38] no, but I don't think gp-dev has the latest code, right? [00:35:44] probably not [00:35:48] i haven't redeployed [00:35:57] or do I just need to pull and restart? [00:36:44] you could try copying the graph, datasource, and datafile to the respective dev-reportcard.wmflabs.org/limn/var/data/* directories and [00:37:02] cool [00:37:05] because if you re-deploy to gp-dev, you'll get the latest code and that's gonna mess with kripke quite a bit [00:37:06] www user is okay? [00:37:13] hehe gotcha [00:37:36] yeah, www user is good [00:37:39] cool [00:38:09] yeah, sorry about this, we're just in the middle of this whole deb/puppet thing [00:38:14] if you have some time in the next few days to push the coloring changes to gp-dev that would be great, but perhaps today isn't the best day because of labs issues [00:38:19] no worries [00:38:36] okay, I'm off to catch a train , be back on soon [15:08:18] morning everyone [15:14:59] hi milimetric [15:15:10] brb in 40m , and we can talk more about the package [15:15:22] cool [15:36:43] back [15:36:48] hi ottomata , dschoon_ [15:36:57] Hiiaayyy [15:37:07] milimetric: did you try dpkg-buildpackage -us -uc ? [15:38:42] I did not [15:38:53] But I'm very confident it works [15:39:06] Did you try it in a VM? [15:39:13] average_drifter^ [15:39:25] milimetric: gonna try it now [15:39:27] I'm scp-ing it now [15:39:33] cool [15:44:16] drdee: good day sir :) [15:50:26] morning! [15:50:37] ottomata: faidon approved the rsync patch [15:50:40] you can merge it [15:50:42] yeehaw! [15:51:03] milimetric, ottomata, could you please make one or two limn charts using the data from kraken / stat1001 [15:51:06] haha, so i'm checking email, here is a mark quote that is funny: [15:51:12] re puppetizing squid confs [15:51:14] > And, adding in Puppet would make configuration changes quite a slow and tedious process, as opposed to how it is now. [15:51:18] hehheh [15:51:24] and email me the links so i can demo it? [15:51:35] regarding squid conf: you forgot to commit your changes [15:51:43] ? [15:51:54] the tab/ cs stuff [15:51:59] it was deployed but not committed [15:52:08] OH! geez, [15:52:30] faidon fixed it [15:52:37] yeah that whole process is kinda janky, thank you faidon! [15:53:23] hey drdee, assign it to me in Asana and add links to the data. I'm almost done coordinating this whole puppetization debianization thing [15:53:52] you have to ask ottomata for the actual links :) [15:54:15] yeah, you'll see, one sec [15:55:32] milimetric; ok created an asana task for you :) [15:58:10] thank you :) [16:17:50] drdee, milimetric [16:17:50] http://stats.wikimedia.org/kraken-public/ [16:18:03] should be synced hourly [16:18:47] we should make that directory all nice clean too [16:19:16] i'll remove my stuff [16:22:41] very cool ottomata [16:29:30] who am i? [16:29:32] grrrrr [16:29:33] not who I want to be. [16:30:16] ahhhgg, can't be who I want to be :( [16:30:20] will try later wah wah [16:37:29] yes! [16:37:36] i have achieved my goals of becoming who I want to be [16:38:03] that was fast [16:40:44] milimetric, my next trick requires ops people in chat rooms to ask them questions. in the meantime, can I help you guys with limn proding? [16:41:15] I'm updating the deployer to work with the new data structure [16:41:35] ottomata: https://github.com/wikimedia/limn/blob/develop/Deb.md [16:41:47] ottomata: have a look at this please, if you follow the steps you will get a limn deb [16:41:51] so you're saying that all of the puppet modules are done, except for something you need help from ops with? [16:42:14] haha [16:42:15] uh huh [16:42:23] no i'm saying that I need them to say I can do something before I do it [16:43:05] hmmm, i think we can use the wikimedia apt for node.js [16:43:13] apt? [16:43:23] http://apt.wikimedia.org/wikimedia/pool/main/n/nodejs/ [16:43:29] ottomata: sure, if you think that's better, please modify the Deb.md in the link above ^ [16:43:30] 0.8.2 ok? [16:43:35] ok w ill [16:43:36] clonging now [16:43:40] thank you [16:43:47] yes, average_drifter, can you make it > 0.8.0 then? [16:43:53] milimetric: no problem [16:43:55] it's like 0.8.19 or something now [16:43:57] cool, thanks [16:44:22] ottomata: does the wikimedia apt have npm packaged as well ? [16:44:34] ok so guys I think the last step is for me to spin up a labs instance to test out both of your work [16:44:58] milimetric: can you tell me what I need to do after limn is installed to start it ? [16:45:04] hmm, average_drrifter, don't think so [16:45:05] I forgot this step [16:45:18] do we need it though? i think we were hoping to not use npm for the .deb install, right? [16:45:25] only to build the package? [16:45:25] on the server right stefan? [16:45:38] ottomata: indeed [16:45:51] aye cool [16:45:54] milimetric: yeah, I have a local limn package, so I install it, and then how do I start limn ? [16:46:08] ottomata: but we have Build-Depends on npm [16:46:19] ottomata: so only the person who wants to build the package needs to have npm [16:46:25] that's fine [16:47:04] average_drifter: you'd need to execute server/server.co [16:47:08] I think I will not modify the readme for building the package [16:47:17] building with chris-leas node.js shoudl be fine [16:47:21] milimetric: is there a command involving supervisor ? [16:47:23] the Deb.md readme [16:47:42] no, server.co is just a coco executable, it can run by itself and go from there [16:48:01] but coco is not globally installed - we tried the local version and it works though right? [16:48:26] milimetric: yep [16:48:51] yeah so like this works average_drifter: [16:48:52] ./node_modules/.bin/coco server/server.co [16:50:04] yeah, ottomata, if ops has a problem with that, we can change it [16:50:24] yeah, i'm not sure what the proper process will be for this [16:50:29] they like to review all debianization stuff too [16:50:33] and usually they want that in gerri [16:50:34] tt [16:50:45] so, i'm not sure [17:04:08] ottomatta, fyi,I just sent you a new invitation to the hangout for today... [17:04:49] oh [17:05:06] oop, sorry, as in not now? [17:05:44] as in now, but I'm guessing you need to check your email for info about the new address... [17:05:50] ahhm [17:06:03] I tried to enter the hangout via google calendar, and it borked me. [17:06:08] hmmm, yeah, hm [17:06:16] ok i'm here at the moment, don't see a new link yet.. [17:06:21] https://plus.google.com/hangouts/_/7f471a23310d2cffe7a2efcc30fb9d7f073109ad?authuser=1 [17:22:53] milimetric: got sec for a limn coloring q? [17:23:05] I got better than a sec [17:23:12] hehe [17:23:16] I got an updated deployer that should fix it once and for all [17:23:23] so this is the one: http://dev-reportcard.wmflabs.org/graphs/new_limn_test [17:23:24] but yes, I also have a sec :) [17:23:32] nice [17:23:40] yeah, it's using the palette wmf_projects [17:23:48] i tried it with and without that palette [17:23:49] so it should get a color [17:23:57] i tried leaving it null [17:24:03] is that the way to go? [17:25:08] no it's straightforward now [17:25:29] if you specify palette at the line-group but no color for line, it'll use a color from the palette [17:25:40] if you specify color for the line, it will use that color [17:25:45] so what are the valid palette names? [17:25:48] cool [17:25:53] wmf_projects, category10, category20 [17:26:07] wmf_projects falls back onto category10 when it can't find a color [17:26:29] hmm [17:26:34] but it'll give you consistent colors for all the labels we hard coded (like Europe, German, Total, Projected, etc.) [17:26:43] the stuff you see on the reportcard basically [17:26:50] so shouldn't the example graph I linked to work? [17:27:04] not deployed yet! [17:27:05] :) [17:27:16] aaah [17:27:17] hehe [17:27:18] nice [17:27:20] you told me to ping you and I've been working on it [17:27:33] just a tiny bit longer [17:27:51] gotcha, forgot about that, sorry. [17:27:59] well i'll await a ping. thanks for working on this [17:30:08] hey, you guys said > 0.8.0 is ok for nodejs? [17:30:25] milimetric: ^^ [17:30:25] my vm has 0.8.1 and npm 1.1.34 [17:30:26] s'ok? [17:31:01] yes ottomata and average_drifter, anything over 8 should be fine afaik [17:31:14] that npm seems a bit old, but let's try it [17:32:19] how will we know? [17:32:26] i'm building right now [17:32:31] your pants will go on fire [17:32:36] oh crap [17:32:49] hm, [17:32:49] npm WARN prefer global coco@0.9.0 should be installed with -g [17:32:49] coco@0.9.0 node_modules/coco [17:32:49] coke build; [17:32:49] /bin/sh: 1: coke: not found [17:33:20] coco has to be installed globally? [17:33:26] or is that an old version of npm problem? [17:41:20] milimetric, average_drifter: [17:41:24] /usr/lib/node_modules/coco/lib/node.js:24 [17:41:25] throw hackTrace(e, code, filename); [17:41:25] ^ [17:41:25] Error: Cannot find module 'underscore.string' [17:42:04] no so instead of coke [17:42:10] we can do node_modules/.bin/coke [17:42:27] that'll use the locally installed coke [17:42:43] I'm sorry my head is spinning right now because kripke is refusing to work [17:42:54] erosen ^ this will hold up deployment for a bit longer [17:43:02] no worries [17:43:07] it's not actually a blocker [17:43:26] i've got the graphs ready, I just wanted to wait for the coloring to work before I put them on the dashbaord [17:45:12] milimetric, you should just get yourself a local vm [17:45:15] precise [17:45:29] virtualbox or something [17:45:41] oh you're linux on your lappy, aren't you? [17:46:00] yeah, I have virtualbox, but in this case it's supervisor that's not passing the environment variable [17:46:10] ahhh, ok [17:46:18] for LIMN_DATA or whatever? [17:46:55] mornin [17:47:00] morning! [17:48:27] morning [17:51:44] yeah ottomata - NODE_ENV and LIMN_PORT are being passed fine through supervisor's environment= thing but not LIMN_DATA [17:51:47] I'm very confused [17:51:51] morning dschoon :) [17:51:57] #does $NODE_PATH describe where [17:52:10] milimetric: I'm havin trouble running coke [17:52:30] NODE_PATH is usually controlled partially by what npm does when it sets things up [17:52:38] (that is, it links binaries into the right places) [17:52:43] you tried node_modules/.bin/coke [17:52:57] exactly. [17:53:08] when it's local, it's node_modules/.bin [17:53:19] when it's global, its NPM_PREFIX/bin [17:53:39] milimetric: yes [17:53:48] milimetric: it says it's missing server.js [17:53:55] milimetric: it's refering to node_modules/coke/lib/server.js [17:54:02] milimetric: how can I tell node where to look for libs ? [17:54:10] I tried NODE_ENV and NODE_PATH to no avail [17:55:21] dschoon can you see what's up with that? I gotta keep wrestling with supervisor [17:56:50] buh? [17:56:53] that makes no sense. [17:57:00] hm [17:57:05] unless cwd isn't being set. [17:57:40] checking for cwd [17:58:07] ottomata, milimetric I think it might be smart to do the replication into gerrit now. [17:58:10] i could set that up today [17:58:21] that way we don't have to hit github for anything [17:59:01] please hold off until we figure out this mess [17:59:05] from github -> gerrit? [17:59:16] I don't want another headache right now, I already have 99 [17:59:32] yes let's wait [17:59:39] sure. [17:59:43] hehe [17:59:46] wouldn't change anyting [17:59:50] milimetric, can I help you with supervisor? [18:00:25] as I looked into it ottomata, my local install decided to do super weird crap [18:00:31] nice! [18:24:39] export NODE_INTESTINES="troubled" [18:27:36] super weird! [18:30:16] oh dschoon - I think I know the problem [18:30:30] var was being served as static right? [18:30:38] yes [18:30:39] so that's how /data datafiles were being served [18:30:43] but now it's a different directory [18:30:43] yes [18:30:46] oh? [18:31:00] because it can be like /var/www/limn-data-for-this-instance [18:31:13] so what do you think is best? [18:31:21] serving that directory as static? [18:31:40] sure, but it depends on how it's structured [18:31:54] structured? [18:31:56] is it always /var/www/limn-data-for-foobar/data ? [18:32:12] because then you can add /var/www/limn-data-for-foobar as a static dir [18:32:14] no LIMN_DATA literally stands for var/data [18:32:15] but if it's: [18:32:31] right, you have to map it to /data [18:32:37] /var/www/limn-data-for-foobar/{datafiles,graphs,datasources,dashboards} [18:32:40] yep [18:32:42] then you have to ... yes [18:32:42] I gotcha [18:32:47] is that possible with static? [18:32:49] yes. [18:32:53] the middleware file has examples. [18:32:57] or i can do it [18:32:59] if you'd like [18:33:15] line 255 [18:33:15] like here: [18:33:15] @use express.static varDir, {maxAge} [18:33:18] yep [18:33:19] instead of that I'd do: [18:33:19] but [18:33:24] @use '/js/limn/test', express.static TEST, {maxAge} [18:33:28] @use 'data', express.static varDir, {maxAge} [18:33:29] right [18:33:33] that mounds TEST at /js/limn/test [18:33:37] *mounts [18:33:40] oh so it's /data not data [18:33:46] mounds :) [18:33:47] right. they're paths. [18:33:52] MOUNDS OF DATAS [18:33:54] cool, doing so now [18:33:56] thanks! [18:33:59] np! [18:34:37] brb a few [18:58:54] ottomata: figured out the dumb problem [18:58:57] it wanted quotes [18:59:02] "./var/data" [18:59:07] because of the slashes [18:59:12] doh [18:59:13] ha [18:59:14] but there were other things at work [18:59:17] that made it a moving target [18:59:23] all fixed, hopefully, deploying now [19:05:01] how do I fix this? [19:05:02] Error: Cannot find module 'underscore.string' [19:08:55] I think that used to be a problem in the old code ottomata, how/where areyou seeing? [19:09:57] erosen: is this an ok time to deploy to gp-dev? [19:10:04] yup [19:10:05] is indeed [19:10:20] ok, dev was ok so I'm gonna start that now [19:10:41] cool [19:10:43] thanks for working on this [19:12:45] root@wmvm:~/deb/build/limn# server/server.co [19:12:45] Failed at: server/server.co [19:12:45] Error: Cannot find module 'underscore.string' [19:14:21] there you go erosen: nice orange color for ya [19:14:22] http://gp-dev.wmflabs.org/graphs/new_limn_test [19:14:33] yay [19:14:59] oo, this one is nice too: http://gp-dev.wmflabs.org/graphs/free_mobile_page_requests_percent_country [19:15:01] thanks [19:17:31] ottomata: this throws a 404 now: http://stats.wikimedia.org/kraken-public/aggregated/mobile/part-r-00000 [19:17:45] yeah, drdee, asked me to change the url [19:17:53] rename the file [19:17:53] uh [19:17:55] what's it now? [19:18:10] http://stats.wikimedia.org/kraken-public/aggregated/mobile/pageview_by_devices_monthly.tsv [19:18:18] cool, gotcha [19:18:31] oh, so underscore.string thing, I'm looking at it [19:18:36] but I think this is more urgent for drdee [19:18:45] yeah no probs [19:27:34] erosen: can we talk ? [19:27:37] erosen: or chat [19:27:41] ya [19:27:44] either way is good [19:28:40] erosen: is this up to date ? https://github.com/wikimedia/metrics/blob/master/pageviews/embr_py/Pageview_definition.png [19:28:48] erosen: because I'm about to implement it [19:29:04] erosen: I'll basically have to simplify my code to adapt to it [19:29:12] the one thing I would expand on, is the way I check for whether the url contains "wikipedia.org" [19:29:18] i didn't really give the full story in the flow chart [19:29:33] i am using a greedy parsing algorithm [19:29:43] where I look for a language code from a known set [19:29:50] erosen: I use a hash for that [19:29:55] if i find it, I chop it off [19:30:21] and then I look for m. or zero. and if I find it i chop it off [19:30:33] I extract everything from http://(stuff).m.wikipedia.org/wiki and then check if it is in a hash with all the accepted stuff [19:30:36] and then i remove .org from the end of whatever is left [19:30:45] cool [19:30:51] erosen: are we doing the same thing ? [19:31:08] well i guess it depends on how you built the hash [19:31:30] is it clear from your code? [19:31:36] mine is about 10 lines of python [19:33:13] 50 lines of C++ http://bit.ly/14X2Ozn [19:33:18] uhm [19:33:40] erosen: can you paste a link to the python code where you do the parsing please ? [19:33:45] ya [19:34:22] https://github.com/wikimedia/metrics/blob/master/pageviews/embr_py/squidrow.py#L230 [19:35:53] self['netloc'] is just the domain with the path stripped off the end [19:37:12] is netloc an array of domain parts ? [19:37:29] for example if the domain was "en.m.wikipedia.org" then netloc would be [19:37:37] yeah, sorry [19:37:39] ["en","m","wikipedia","org"] ? [19:37:44] exactly [19:38:05] it is defined right above netloc_parsed: https://github.com/wikimedia/metrics/blob/master/pageviews/embr_py/squidrow.py#L225 [19:40:13] drdee, ottomata: http://test-reportcard.wmflabs.org/graphs/pageviews_mobile_by_manufacturer [19:40:28] erosen: I think I'm not counting the zero urls [19:40:35] erosen: how does a zero url look like ? [19:40:41] it shouldn't matter much, as they are very rare [19:40:48] ok [19:40:50] but they have the same syntax as a mobile domain [19:41:02] so en.zero.wikipedia.org right ? [19:41:02] except m. is replaced by zero. [19:41:05] yup [19:41:17] it might not work for you, because it only works for certain ip ranges [19:41:19] ok average_drifter, I'm sorry, I'm back to normal [19:41:22] it works in the office though [19:41:24] cooooOOl! [19:41:33] did dschoon help you with that coke problem? [19:41:35] milimetric: you're back to normal ? [19:41:36] naw [19:41:37] still having it [19:41:40] milimetric: cool [19:41:46] how can i help? [19:42:06] # server/server.co [19:42:06] Failed at: server/server.co [19:42:06] Error: Cannot find module 'underscore.string' [19:42:18] ok, let's see, we should build the deb again [19:42:34] erosen: so what I do is I just throw in a hash http://en.m.wikipedia.org/wiki , http://ja.m.wikipedia.org/wiki etc etc [19:42:35] ottomata: does node_modules/underscore.string exist? [19:42:42] yes [19:43:00]