[15:10:11] morning! [15:15:36] moooooooooooooorning ottomata, milimetric, average_drifter [15:15:40] hi drdee [15:15:46] whazaaaa! [15:15:54] drdee: morning :) [15:15:55] i got some exciting ideas! [15:16:00] whizeeeee! [15:16:10] yooo morning [15:16:14] yea? [15:16:24] average_drifter: we Skype first, then milimetric [15:16:29] whuuzuuuuuuuu [15:16:36] !!! [15:16:36] k :) [15:16:57] ottomata; you were totally right about blog traffic [15:17:38] so can just set that up whatever you think is easiest (awk, manually, sed) don't really care, kraken would be nice but also not a requirement [15:17:49] let's just start collecting the page views for the blog and get it done [15:18:22] ok [15:18:39] i think the data will be not big enough to justify putting into kraken [15:18:45] at least, not big enough for hadoop [15:19:09] k [15:22:00] isn't the point of kraken to be a centralizing influence also, not just a big data handler? [15:23:24] drdee: tried to call you [15:23:50] yup [15:23:52] hol don [15:23:54] hold on [15:23:54] ok [15:33:46] so,, drdee, how can I help with the blog traffic? I don't really remember what needs to be done…we are goign to use awk or something instead of udp-filter -o? [15:34:26] just anything to start collecting the data :) [15:34:45] just a filter to catch the log lines for blog? [15:34:56] udp-filter -d blog.wikimedia ? [15:40:28] sounds good ! [15:49:50] oh right but we need to filter on the unsampled stream, right? [15:55:14] drdee, was there a reason we couldn't just add it to the list of projects in filter.c and recompile? [15:55:37] (sorry, the only knowledge about the details I have of this project are from the chat we had here a week or two ago, and I don't remember much) [15:55:56] yes because it required all the business logic which you opposed :D [15:56:18] blog != wiki [15:56:31] i'm am for the minimal amount of change possible :) [15:56:39] that's why i said you were right [15:56:42] OHHH right [15:56:42] right [15:56:45] ok I remember [15:56:46] RIGHT! [15:56:57] cause it does stuff with the paths too, riiiight ok [15:57:27] so we need filter for blog on unsampled stream [15:58:13] ergh, I don't really have a good place to run this [15:58:14] hm [16:06:24] :D [16:22:41] hmmmmmmmmm [16:22:42] drdee [16:22:51] what if I pipe the blog stuff into kafka, but then consume into a file on stat1? [16:22:52] :D [16:24:21] sure! [16:24:28] as long as we have the data [16:25:09] hmmmm ok! [16:25:27] i think I still have packet loss problems though :( [16:27:59] ohhhh nooooo….. [16:39:26] yeah, i thought it was good because i didn't see any dropped packets in /proc/net/udp soon like I had before [16:39:36] and the packet-loss.log wasn't working for some reason, i think I just fixed that [16:39:37] so we will see [16:56:21] fingers crossed [17:49:46] mornin [17:50:07] morning [17:50:57] morning dschoon [17:51:46] milimetric: man, MoMath looks *great* [17:57:45] https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90 [18:01:18] ottomata ^^ [19:23:11] if you guys haven't read it yet, Brandon's email is hilarious [19:23:32] http://www.cardsagainsthumanity.com/holidaystats/ [19:24:37] what. [19:24:42] i can get an island for $55k?! [19:25:09] (yay wikipedia) [19:25:49] :D [19:26:03] i like how their postage outstripped production costs by like 5:3 [19:31:02] lunch, i think [20:01:55] ottomata, dschoon, drdee: would appreciate it if one of you cld "apt-get install zpubsub python-zmq" on stat1 [20:02:04] oh sorry, yeah i'm in ops meeting, can do [20:02:05] or even better, sudo apt-get [20:02:21] is it worth puppetizing that? [20:02:29] ottomata: i'm the one who's sorry -- i just sent you the request a little bit ago. it's urgent because i'm a dork and forgot to do it earlier. [20:02:53] naw, its cool [20:02:56] ottomata: you could include the eventlogging class i added in that change you merged earlier [20:02:58] i just did it [20:03:01] it includes these packages [20:03:03] yay! :) thanks. [21:25:26] drdee, did you get that script to run (ops meeting over!) [21:29:00] hey dschoon, you around? [21:29:09] yep. [21:29:14] what's up? [21:29:14] idea. [21:29:23] what if we made a udp2log like storm spout? [21:29:46] is that really different from what we're doing now? [21:29:48] sorry in chat too [21:29:49] what's the advantage? [21:29:50] hang on, be back in a sec [21:29:53] sure [21:30:28] i actually have an errand i need to run today before this places closes [21:30:34] can we chat about it in an hour or so? [21:30:57] naw i'll be gone, lets talk tomorrow [21:31:03] okay. [21:31:10] brb again, boo [21:31:20] but in ~1h [21:34:12] yo [21:34:23] ottomata, working on the pig script [21:34:36] did you use a tab as separator between the kafka offset and the hostname? [22:01:25] hey [22:01:39] i did not do that, but, the hadoop consumer script might [22:01:52] k [22:37:55] ping ottomata [22:48:39] milimetric: do you have any d3 world map boiler plate around? I need to make a one off map for the mobile team. [22:49:04] the one david made for editors [22:49:16] did he ever show you? [22:50:23] hmm [22:50:26] from way back? [22:50:37] i know of one that fabian made [22:50:48] that sort of answers my question, actually, assuming I can find it [22:51:06] oh yeah fabian [22:51:11] i'll look for it too [22:55:38] I think this is at least part of the source: https://github.com/wikimedia/limn/blob/master/src/main-geo.co [22:55:40] but ... in coco [22:55:49] yeah [22:55:51] that's cool [22:55:52] awesome [22:55:56] thanks for finding that [22:56:40] this might be easier though: http://bl.ocks.org/3306362 [22:56:58] awesome [22:57:05] that might be better [23:02:55] ok, erosen, you're looking for countries right? [23:02:56] http://bl.ocks.org/4121134 [23:03:03] yup [23:03:05] that might be even better [23:03:18] because the other one seemed to be a US-specific projection [23:04:34] there's an svg path object for every country, filled with a color generated from the name of the country [23:05:03] I'm mocking up the sankey diagram now, if you want I can get this up next (just send me a link to the data) [23:05:30] i might try my hand at it for fun [23:05:36] but thanks for the pointers [23:06:03] yeah, it looks like there's active work from mbostock and team on this kind of vis. [23:41:27] back [23:42:33] milimetric, erosen: there's also https://github.com/mbostock/topojson [23:42:42] which is map-related, but of different utility [23:43:34] nice [23:56:29] drdee, you about? [23:56:42] yes [23:57:45] dschoon ^^