[03:55:52] hey [03:55:55] drdee, you there? [03:56:03] yes [03:56:13] why is kafka running on vanadium? [03:57:07] not sure [03:57:25] good times. [03:57:36] since when is it running? [03:57:49] today, it seems [03:57:58] could it be puppet related? [03:58:08] (that it somehow got included) [03:58:16] yes, maybe i'm at fault. i'll check [03:58:22] re: awk, yes, that's a good idea [03:58:41] i never thought i'd live to see you chide me for over-engineering something :) [03:58:51] we didn't talk about having kafka on vanadium [03:58:59] so expect some mistake [03:59:08] i mean I expect [03:59:24] its invocation is a marvel of engineering simplicity and elegance: [03:59:26] java -Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:/usr/lib/kafka/bin/kafka-console-consumer-log4j.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -cp :/usr/lib/kafka/bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:/usr/lib/kafka/bin/../project/boot/scala-2.8.0/lib/scala-library.jar:/usr/lib/kafka/bin/../core [03:59:27] /target/scala_2.8.0/kafka-0.7.2.jar:/usr/lib/kafka/bin/../core/lib/*.jar:/usr/lib/kafka/bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:/usr/lib/kafka/bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:/usr/lib/kafka/bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:/usr/lib/kafka/bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:/usr/lib/kafka/bin/../core/lib_managed/scala_2.8.0/co [03:59:27] mpile/zkclient-0.1.jar:/usr/lib/kafka/bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar kafka.consumer.ConsoleConsumer --group blog0 --topic webrequest-blog --from-beginning --zookeeper analytics1023.eqiad.wmnet,analytics1024.eqiad.wmnet,analytics1025.eqiad.wmnet [03:59:48] yikes [03:59:50] this should solve all of the complex difficulties we had with udp logging [04:00:32] although it might be that ottomata did this, i see webrequest-blog [04:00:43] TL;DR [04:00:48] i gave up halfway through chapter two [04:00:57] and that means it is collectiong unsampled data for blog.wikimedia.org [04:01:14] i have to ask ottomata, [04:01:21] that's a shockingly small classpath! [04:01:35] you should see the shit the hadoop jars demand. [04:01:47] (also, most of that is due to scala, heh) [04:02:02] i'll quote my fourth-favorite dutchman, after you (of course), krinkle and siebrand: [04:02:05] "Simplicity is prerequisite for reliability." [04:02:22] well. [04:02:39] ohoh philosophical debate [04:02:40] i'm sure that process is a fork off of something like $ ./kafka-start.sh [04:02:43] i am outta here! [04:02:53] so it's up to you whether that means simplicity or complexity. [04:03:38] oh, if it's hidden away under a shell script then it's fine [04:03:50] we'll just have to remember to run ./kafka-fix.sh if anything ever goes wrong [04:03:52] be honest, drdee. you're bolting because you've had munchkin vomit on your shirt for the last three hours. [04:03:59] i think it's called ./unfuck.sh [04:04:24] huh [04:04:27] it's just one line [04:04:32] $ cat unfuck.sh [04:04:32] #!/bin/bash [04:04:32] sudo killall udp2log [04:04:34] apt-get remove kafka && apt-get install udplog [04:04:39] ;) [04:05:12] you read rich hickey's simplicity deck? [04:05:33] yes [04:05:49] what more: understood it, too [04:05:56] really good stuff. [04:06:42] bbl, ttyl, etc. [04:07:16] much love~ [04:11:42] what is rich hickey's simplicity deck? [04:14:01] http://www.infoq.com/presentations/Simple-Made-Easy [04:14:13] ty [04:35:43] * fifth favorite [04:35:54] forgot roan. [04:47:03] man i am dropping like a brick [04:57:41] it's midnight! [11:00:52] hashar: hey Antoine [11:01:00] hashar: build1 has no more disk space [11:01:08] hashar: can we have some diskspace there please ? [11:01:13] hello [11:01:16] what is build1 ? [11:01:21] build1.pmtpa.wmflabs [11:01:32] ha [11:01:37] it shares the /home directory with build2.pmtpa.wmflabs [11:01:37] I am not an op can't do that :-] [11:01:39] and with krupke [11:02:07] what is the /home ? [11:02:15] is that the globally shared /home ? [11:02:32] hashar: I can just say it's shared between build1, build2 and krupke [11:02:41] hashar: not sure if globally [11:02:44] probably a mount to some other filesystem [11:02:59] df -h /home [11:03:03] will tell you :) [11:03:08] spetrea@build1:~$ df -h /home [11:03:09] Filesystem Size Used Avail Use% Mounted on [11:03:09] labs-nfs1:/export/home/analytics 18G 17G 141M 100% /home [11:03:17] so that is readonly now [11:03:20] yes [11:03:29] the labs people have migrated the /home to glusterfs [11:03:45] you have to reboot the instance to be able to use the new /home (data have been migrated) [11:03:51] so just reboot them all :-] [11:03:53] also [11:03:59] YOU SHOULD NOT USE /home :-] [11:04:04] instead use /data/project [11:04:08] which is per project [11:04:17] /home is shared by all projects [11:05:10] alright [11:05:14] I'll reboot then [11:05:43] rebooting... [14:51:59] morning guys [14:52:45] drdee: hi :) [14:53:03] just talked with Erik on g+ [14:53:10] ok [14:53:13] I updated him with most of the developments [14:53:23] nice! [14:53:29] morning [14:53:43] moooooorning! [15:21:42] today I am going to try to work all these things: [15:21:42] 1. setting up a flask site for Ryan F on stat1001 [15:21:42] 2. LDAP groups + hue. Ryan L merged in my changes. [15:21:42] 3. Getting packet loss nagios alerts (I talked to notpeter and know how to do this without angering ops) [15:21:42] 4. Dear Cron, STOP EMAILING ME [15:21:42] 5. run another terasort bench [15:23:58] great! [15:25:20] also, so far, /proc/net/udp reports 0 dropped udp2log packets since I started it up yesterday [15:28:16] woot [15:29:26] is locke the producer of squid logs ? [15:29:44] hm, no, but i'm not sure what you are asking [15:31:23] ottomata: just forwarded an e-mail Erik sent me about packet loss [15:31:36] ottomata: the e-mail describes the solution when this occured back in 2011 [15:32:10] I am unsure whether this would fix it now, I just thought it might be of interest [15:32:52] thanks, yeah, hehe, in this case I fixed packet loss by adding an awk script :p [15:33:06] but yeah that one there looks kind of expensive [15:33:13] regex matching for every single log line [15:33:43] have you guys ever done a git merge -s ours a_Branch_You_Want_To_Ignore_But_Keep_History_For? [15:33:47] because it's awesome [15:36:28] -s ? no! [15:36:45] oh strategy [15:37:27] how does that work? put the history in the current branch but without making real changes? [15:37:45] I use --squash for that [15:37:51] maybe -s is --squash ? [15:38:27] * average_drifter goes back to the bug [15:39:47] the manual sayeth: "This resolves any number of heads, but the resulting tree of the merge is always that of the current branch head, effectively ignoring all changes from all other branches. It is meant to be used to supersede old development history of side branches. Note that this is different from the -Xours option to the recursive merge strategy." [15:39:57] so basically, yes ottomata, what you said [15:40:23] it makes that branch an ancestor of my current branch but leaves everything exactly how I have it [15:41:49] weird [15:42:04] it was exactly what I needed for Limn because we basically abandoned a couple of branches [15:48:41] i like the verb 'to complect' a lot [17:10:17] i recently cut myself on something so sharp i didn't notice i cut myself. [17:10:34] this is problematic precisely because i have no idea what cut me. [17:10:45] magic mystery knives == bad [17:12:42] hehe [17:14:13] christ, it's 37F today. [17:14:16] who let that happen? [17:17:11] that looks like 0x37F [17:17:30] must be hex [17:18:41] i wish [17:18:54] hm. [17:18:55] maybe not. [17:19:01] then it'd be like 895 degrees [17:19:14] though without a scale. [17:19:31] i'd be totally down with 0x37 degrees F though [17:19:35] that's like 55 :) [17:19:50] aiight. heading in. [17:31:47] Ping robla, Skype? [17:58:36] haaangout [17:58:47] https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90 [17:59:09] coming [18:00:41] still talking to robla [18:29:51] heading upstairs [18:33:23] erosen, drdee: you guys hanging out? [18:33:34] we are indeed [18:33:43] dschoon is on the 6th floor, too [18:33:46] sup DarTar -- i'm in R62 if you're in the office [18:34:01] kicking it [18:34:06] k, will be there in a sec [18:45:12] http://meta.wikimedia.org/wiki/Research:Metrics [18:49:40] https://office.wikimedia.org/wiki/Editor_Engagement_Experiments/Analytics_Databases [18:49:54] dschoon: can you copy it to the hangout ^^ [18:55:10] ottomata, what version of CDH are we running? 4.1.2? or 4.1? something else? [18:57:51] ottomata, it seems that we are running 4.1.1 [18:58:01] could you upgrade to 4.1.2? that would fix my bug [19:03:05] ottomata ^^ [19:03:47] 4.1.1 i think [19:03:49] sorry was making lunch [19:03:50] um [19:03:57] Hadoop 2.0.0-cdh4.1.1 [19:03:59] you can find out by running [19:04:00] hadoop version [19:11:30] can you upgrade to cdh 4.1.2? [19:40:18] ja we can, want to do that sometime, do you have a need for it right now? [19:40:29] oh [19:40:31] i see the response [19:40:31] You are probably hitting a Oozie ShareLib problem in CDH4.1, fixed in CDH4.1.2 in the Oozie app. [19:41:05] sure, i just finished a response myself [19:43:05] ty SIR! [19:45:27] haha [19:45:36] i like how andrew gets things done before you finish asking. [20:08:29] um, drdee, i think we are actually using 4.1.2…at least, it is already on some of the nodes, i think just an01 and some of the ciscos were on 4.1.1…double checking now [20:08:55] an01 was on 4.1.1, iIRC [20:10:03] yeah it was [20:10:06] its on 4.1.2 now [20:10:13] aight [20:10:21] i'll try shortly [20:10:29] ah, and an23- an27 [20:10:31] are on 4.1.1 [20:10:33] upgrading now [20:10:44] is there a way of preventing this in the future? [20:10:54] i mean automated way [20:11:09] you can manually specify versions in puppet [20:11:11] so yeah we could [20:11:22] i think its like this because the machines were installed at different times, and different packages were available [20:11:31] if this was in prod, ops wouldn't let us use cloudera's apt repo [20:11:33] so this wouldn't happen [20:11:48] ok [20:17:37] drdee, upgrade done. [20:17:42] thanks! [20:31:05] drdee: have to reschedule the pv meeting [20:31:11] ok [20:57:56] brb foods [21:56:17] ahhh phooey [21:56:19] dropped packets [21:57:54] :( [21:58:11] hmm, packet-loss.log seems low enough….i'm just seeing dropped packets in /proc/net/udp [22:14:48] dschoon: you around? [23:17:14] dschoon: you mentioned you wanted to experiment with some log parsing storage. I'm not sure if what you wanted exactly, but I thought this might be useful: https://github.com/embr/squidpy [23:17:28] cool! i'll check it out [23:17:30] it's just a little squid log line wrapper class [23:17:34] *nod* [23:20:36] Yeah, I wrote a simple order-based parser for my avro experimentation [23:20:47] though I now notice that I didn't commit that anywhere, heh [23:21:01] hehe [23:21:53] (on the todo list it goes)