[00:26:21] milimetric: but that was fixed as well [00:26:30] fixed where [00:26:48] settings.py doesn't get committed (thank god because it has prod passwords) [00:27:06] so it has to be manually updated on the server [00:27:23] I should've probably +2 +2 -ed it and made it so only otto can merge, then written a note [00:27:43] i made a separate commit to fix that [00:27:54] to fix what? [00:28:05] please say you didn't add a settings.py [00:28:06] to have settings.py have a version variable [00:28:22] no settings.py.example is all set [00:28:24] no, i fixed the template in puppet [00:28:29] oh! [00:28:31] i see [00:28:42] so that must not be merged then [00:28:42] so not sure why this does not work [00:28:47] it was [00:28:51] i checked with ottomata [00:29:16] oh that's very weird then [00:29:24] but that definitely looks like what the prob is [00:29:32] lemme see if i can read settings.py on stat1001 [00:30:11] nope, permission denied [00:33:51] back [00:35:26] yep [00:35:38] so i couldn't read the settings.py file on stat1001 [00:35:49] but that's the next step [13:06:56] moooooorning [13:29:40] morning ottomata [13:30:19] morning! [13:32:25] talking to paravoid [13:33:36] oh? should I join? [13:34:33] drdee, looking at some of the packages in your list, there are some that you have listed (like snappy at least) that ship with cdh4 [13:34:40] oh [13:34:44] but there are apt debs too [13:34:44] hm [13:42:09] hey, drdee, metrics api ok? [13:42:29] yup [13:51:11] lk [13:51:36] what's up with faidon? should I join your convo? also, alex (new ops guy) has given me a cdh4 hadoop review, so I have some work I can do on that [13:51:42] shoudl I work on Kafka or Hadoop? [13:51:54] most of hte Hadoop stuff will be responding to his thing, so it probably won't take that long [13:52:09] drdee^ [13:52:31] he just fixed a couple of syntax errors, right? [13:54:53] alex? maybe, and has questions about things [13:55:01] he wants to add unit tests, which is cool [13:55:06] i'd like to learn how to do that for puppet modules [14:10:58] ottomata: brain bounce [14:10:59] ? [14:17:37] sure [14:17:42] actually [14:17:46] lemme finish responding to alex's comments first [14:17:48] 10 minutes [14:17:54] aiiiiiiigggghhht [14:42:59] sorry, drdee, almost ready, gotta pee. this has been way longer than 10 minutes [14:47:15] ottomata:https://plus.google.com/hangouts/_/36acd5c9da2a2749cdea5543b414d6b46fcdfeac [14:59:09] uhng, sorry drdee [14:59:16] cafe internet stopped working, spent a while getting back on it [14:59:19] on phone internet now [14:59:35] drdee, i'm in the batcave [15:01:14] [travis-ci] master/382de50 (#134 by milimetric): The build has errored. http://travis-ci.org/wikimedia/limn/builds/7189981 [15:02:12] boo [15:02:17] boo travis boo [15:02:20] [travis-ci] develop/24877dd (#134 by milimetric): The build is still failing. http://travis-ci.org/wikimedia/limn/builds/7189982 [15:02:22] boo! [15:02:43] * milimetric feels insane for yelling at robots [15:23:11] drdee, I don't see any jakarta-log4j package [15:23:17] but i do see liblog4j1.2-java [15:23:17] which is [15:23:21] 1.2.16-3ubuntu1 [15:23:25] not quite the exact version [15:23:27] but would probably work [15:23:56] uh, I don't see libsnappy-java either? [15:23:59] what repos are you seeing these in? [15:24:26] libslf4j-java matches though [15:24:28] i could try that one [15:26:43] i just used the mediawiki vagrant vm [15:26:49] and did apt-get install [15:32:07] hmm, weird [15:32:21] dunno [15:32:39] uh, i don't even see that jakarat one on mediawiki vagrant vm [15:35:49] E: Unable to locate package jakarta-log4j [15:35:51] drdee ^ [15:35:54] dunno where you got this stuff :p [15:35:55] :) [16:08:38] lunching, back in a bit [16:08:52] guys, can you move all your mingle cards in the right place? [16:33:39] What version of varnish will/are you using? 3.0.2? [16:34:15] we are using a slightly custom built [16:34:18] 1 sec [16:35:24] 3.0.3 but custom: https://gerrit.wikimedia.org/r/gitweb?p=operations/debs/varnish.git;a=tree;h=HEAD;hb=HEAD [16:35:57] probably best to use that source code [16:36:08] okay, thanks [16:45:05] drdee: I am completely confused about the silly version string [16:45:18] whatsup? [16:45:25] and would also like to point out that what we thought was a 5 minute thing has now taken the server down and cost us a total of maybe 2 or 3 hours [16:46:34] well, so you were absolutely right drdee, it executes fine [16:46:39] i ran it from the shell, and I don't have sudo [16:46:57] and your exact code gets the version string exactly how it does on my local [16:47:08] and it's pretty cool btw [16:47:44] so the only thing I can think is that maybe settings.py wasn't really updated properly by puppet [16:48:37] ottomata: can you cat settings.py and paste the part up to this line: [16:48:37] version = 'Unknown version' [16:49:30] it's in /a/e3/E3Analysis/user_metrics/config/settings.py [16:49:55] or, rather, make sure it has the version, err = p.communicate() line in it... [16:51:02] wow, crazy that we only funded $350 of grant money to China [16:51:17] that could be like... an explosive change at some point [16:51:25] yeah, pretty cool to look at [17:20:11] milimetric: version issue resolved? [17:20:23] and can you and erosen do a mini stress test on metrics.wikimedia.org ? [17:28:00] no, the version issue's not resolved drdee [17:28:33] erosen: i'm not sure what's quicker - what's the link to that apache load tester drdee? [17:28:33] or just doing it manually [17:28:52] i suspect he meant do it manually [17:32:35] milimetric: i've got to catch a BART to get to the office, but I should make it in time for the showcase [17:52:52] milimetric; ready to show case the mobile page view stuff? [17:53:00] yes [17:53:04] ottomata: ready to showcase the packet loss monitoring per dc/role? [17:53:04] i have a file of results for april [17:53:08] cool! [17:53:11] for just the 3rd party apps [17:53:16] only 3 of them were found in the sampled logs [18:00:28] drde, yup! [18:02:46] ottomata sprint demo [18:02:52] https://plus.google.com/hangouts/_/fe204c66797be79ba879f581228898105b0efde5 [18:03:12] i'm there! on the one from the email! [19:41:23] ottomata, any news from kafka debian? [19:41:42] ha, not much since i've been in meetings for the last 2.5 hours :) [19:41:46] but i'm working on it... [19:41:47] (as is, did the trick work?) [19:41:50] i thikn so [19:41:55] i was successful with the symlink [19:42:00] sweet [19:42:02] i was trying to add /usr/share/maven-repo as a resolver [19:42:05] couldn't make that work [19:42:14] i'm going through and seeing how many more things I can symlink [19:42:20] ok [19:42:36] at some point i'm going to try to take the built release [19:42:39] OOOOO!!! [19:42:42] what if. [19:43:04] what if debian/rules symlinks /usr/share/kafka/libs/*.jar to their /usr/share/java/ files? [19:43:08] the ones installed by debian? [19:43:28] that way I don't have to mess with classpath more than I have to [19:43:30] hmmm [19:44:24] that sounds like a very sweet solution [19:44:28] and way less hacky [19:48:09] i think i still have to do what we are talking about [19:48:13] for the build [19:48:29] including the .ivy2 stuff and symlinking the .jars there too [19:48:41] that way we can be sure that we are building kafka with the same jars that will be used at runtime, right? [19:49:05] true [19:49:52] i'm making progress, but I feel like I understand about 10% of what is going on right now [19:51:19] uhh, not sure what to do with scala compiler, it does come with the deb scala package, but I don't see any .jars in .ivy2 [19:57:31] i think those are installed somewhere else [19:58:42] hmmm ~/.sbt? [19:59:25] /usr/share/scale ? [19:59:44] naw, the ./sbt installs it somehow [19:59:52] it would be in /usr/share from the .deb [19:59:56] but I need to know where to symlink it from [20:01:00] ~/.ivy2/cache/org.scala-lang [20:02:45] naw, i don't have that [20:02:47] i mean [20:02:48] i have it [20:02:50] but no .jars there [20:02:57] i have ~/.sbt/boot/scala-2.9.2/lib [20:05:23] true, i only have scrs [20:05:28] but maybe that's okay [20:06:14] well, the kafka release ships with scala-compiler.jar [20:06:18] so I have to do sometihng with it [20:06:18] hmm [20:06:25] is there a -Dsbt.home I can use or something? [20:07:16] but we should be able to trust right [20:07:21] ? [20:07:30] i'm trying to make it so this thing will build 100% offline, right? [20:07:34] right [20:07:37] heyyyaaa i htink -D sbt.home works! [20:07:44] wow [20:11:04] yargh, but it creates a ~/.sbt directory [20:11:04] even if I don't want it to [20:11:27] milimetric; can you https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc for a sec? [21:56:46] drdee: still around [21:56:57] cool [21:57:01] I missed your message (looking at nasty iterator logic David left for me as a surprise?) [21:57:07] haha [21:57:10] I mean! [21:57:14] hangout? [21:57:21] for what you needed ,not me [21:59:40] i'm in the hangout if you still need [21:59:42] yurik was asking us if we could find a couple of mobile requests to help him debug a wikipedia zero problem [22:05:05] i can do that [22:24:02] erosen:cool! [22:31:32] drdee: is yuri on IRC? [22:31:42] yurik, yes