[08:59:18] (PS1) Ori.livneh: Ensure metadata is emitted every send_metadata_interval [analytics/statsd-ganglia] - https://gerrit.wikimedia.org/r/83393 [08:59:34] (PS2) Ori.livneh: Ensure metadata is emitted every send_metadata_interval [analytics/statsd-ganglia] - https://gerrit.wikimedia.org/r/83393 [08:59:40] (CR) Ori.livneh: [C: 2 V: 2] Ensure metadata is emitted every send_metadata_interval [analytics/statsd-ganglia] - https://gerrit.wikimedia.org/r/83393 (owner: Ori.livneh) [14:37:30] helllooo Snaps_! [14:38:06] I'm building librdkafka + varnishkafka with the compression support [14:38:16] I don't see a related varnishkafka commit [14:38:23] am I missing something? [14:38:31] how do you configure varnishkafka to use compression? [14:43:18] all librdkafka configuration properties are available by fallthru in varnishkafka, so in this case there is no need to do anything with varnishkafka, not even upgrade it (as long as its linked dynamically with librdkafka) [14:43:53] compression.codec = snappy | gzip | none [14:45:10] ahhh, awesome [15:06:59] hmm, Snaps_, with compression on, how often are batches of messages sent out? should be the usual rate, right? [15:07:10] Its configurable! [15:07:13] right [15:07:18] so, what i'm trying now [15:07:35] i've got a console consumer reading topic varnish1 [15:07:51] without compression.codec = snappy, varnishkafka produces to kafka and I see the messages in the consumer [15:07:57] but with compression = snappy [15:08:13] i don't see anything going wrong (haven't looked very hard yet), but I am not getting the messages at the consumer [15:08:36] "compression = snappy" shouldnt be a valid configuration statement and varnishkafka will refuse to start [15:08:46] yeah sorry [15:08:49] compression.codec [15:08:54] compression.codec = snappy [15:09:10] ah, sorry, I read the first one was "with", not "without" [15:09:53] i think varnishkafka is working with compression.codec = snappy [15:10:00] but i'm not getting anything in the consumer so I can't be sure [15:10:12] debug level logging reports no produce errors [15:10:27] ohhh [15:10:37] ? [15:10:39] i am getting failed produce requests at the broker (looking in jmx now) [15:10:59] add debug = msg,broker to .conf [15:11:03] and sset log.level = 7 [15:11:41] looks normal i think [15:11:47] does the broker say what is wrong? [15:12:13] failed produce requests on the broker would indicate that the compressed messages are erronously formatted [15:12:18] ahhh ha [15:12:18] java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy [15:12:23] aha! [15:12:32] my kafka isn't built with snappy? or is that a java deb package? bwrrrrp [15:12:47] dunno :) [15:12:54] libsnappy-java :) [15:13:09] oh its installed, hm [15:13:20] oh i wonder if it has to be on the classpath [15:13:20] hmm [15:14:20] hmm it is [15:27:30] Snaps_: when you test kafka, do you just run their shipped bin/kafka-server-start.sh script? [15:27:45] or do you use our debianization? [15:28:59] I use the shipped one [15:29:05] executing from the source dir [15:29:25] aye k [16:15:42] ottomata: success? [16:15:48] naw, in meeting now [16:15:50] not sure what is up [16:16:00] the snappy lib is in kafka's CP [16:16:01] so dunno [17:00:15] (Abandoned) Matthias Mullie: Generate data for Echo preferences [analytics/limn-ee-data] - https://gerrit.wikimedia.org/r/71999 (owner: Matthias Mullie) [18:39:15] yoo so qchris, you wanna check out this kafka snappy thing with me? [18:39:57] Sure. [18:40:03] Let me boot the google machine. [18:40:39] ah well we can IRC this one [18:40:48] Even better :-) [18:41:04] Which machine does it run on? [18:41:11] kraken-kafka.pmtpa.wmflabs [18:42:50] so qchris [18:43:02] hmm, actually, maybe hangout would be good [18:43:07] i can show you a buncha terminals at once :p [18:43:14] Hehe. [18:43:18] I knew that :-) [18:55:27] hey Snaps_, what java are you using? [18:56:09] for kafka testing [18:56:10] ? [18:56:33] java version "1.7.0_25" [18:56:33] OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.13.04.2) [18:56:33] OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode) [18:56:40] k danke [18:57:51] qchris: https://gist.github.com/ottomata/6499933 [19:04:02] ottomata: thanks for the merges [19:20:09] yup! [19:20:40] Snaps_: can you tell what version of snappy-java you are using? [19:21:17] 1.0.5-2 [19:21:34] the ubuntu package, that is, but its possible that kafka downloads and builds its own [19:25:07] hm, 1.0.5-2 is the ubuntu package? lemme see, I have 1.0.4.1 [19:25:21] but i betcha sbt does download its own [19:25:55] hmm, nope [19:25:56] well i mean [19:25:58] it does [19:26:03] but the version is pinned at 1.0.4.1 [19:26:05] "org.xerial.snappy" % "snappy-java" % "1.0.4.1", [19:26:07] which is what we have [19:26:08] hm. [19:28:21] ottomata: /home/qchris/snappy-java-1.0.4.1.jar [19:37:00] ottomata: /tmp/snappy-1.0.4.1-libsnappyjava.so [19:47:10] hey Snaps_, [19:47:32] when you run the console-consumer [19:47:43] do you get a /tmp/snappy-1.http://0.4.1-libsnappyjava.so/ [19:47:44] file? [19:48:34] ottomata: You mean '/tmp/snappy-1.0.4.1-libsnappyjava.so' in the question to Snaps_ ? [19:49:07] yes [19:50:16] yup [19:50:35] I do [19:50:39] hmm, ok [19:55:19] Snaps_: so you got 1.0.5 from the libsnappy-java package? [19:56:09] scratch that, libsnappy-java isnt installed [19:56:41] ohh [19:56:42] hm [19:57:21] hm, do you know where you got 1.0.5 from? [19:58:30] thats the c++ library package. [19:58:35] sorry about that [19:59:33] oh ok [20:01:34] it seems to have been downloaded using maven [20:01:52] ./core/target/scala-2.8.0/cache/core/global/assembly/5a345cc879d7dc549407387293e5b1e74a4998d3/META-INF/maven/org.xerial.snappy/snappy-java/LICENSE_snappy-java-1.0.4.1 [20:03:23] ok hm [20:03:28] can you ls -lh that [20:03:30] how big isit? [20:03:31] or [20:03:33] even better [20:03:37] does that .jar have the .so included in it? [20:03:47] Snaps_: ^ [20:07:17] ottomata: http://pastebin.com/ytYDTu1f [20:12:52] ottomata: i built openjdk 7 for osx this weekend [20:12:58] if you reed it let me know [20:13:11] YES I COULD NOT FIND A installer for OSX!!! [20:13:18] so i had to built it mysefl [20:22:05] * ori-l awards drdee 9,000 mingle points [20:29:05] haha [20:29:15] haha, thanks drdee, but i do everything in my vm so as not to mess up my mac :p [20:47:15] hey guys https://mingle.corp.wikimedia.org/projects/analytics/cards?favorite_id=1072&view=Backlog+Grooming [20:47:36] great, thanks qchris [20:47:44] oops, i didnt' mean to close our chat, but i guess we're good for now thne [20:47:58] we know what's wrong, hopefully alex will think of a good solution here, especially since he built the kafka package [20:47:59] ottomata: I didn't do anything. [20:48:08] haa, yes you did! i we figured that out together! [20:48:25] Whatever. [20:48:40] Hopefully he finds a good solution. Yes. [20:48:52] yeah [20:49:06] Then becak to monitoring geowiki :-) [20:49:09] yeah, maybe he'll just say we can include it with the kafka .dev? :p [20:49:16] s/becak/back/ [20:49:29] :-) Yes. Maybe. [20:49:40] But that looks a bit weird to me. [20:50:15] We'll see :-) [22:10:17] average are you here?