[00:13:29] ? [13:51:05] morning qchris [13:51:06] batcave? [13:52:09] morning milimetric [13:52:15] yo qchris_away [13:53:15] morning average [13:57:45] morning drdee [13:58:42] i can't access the batcave [13:58:44] like seriously [13:58:49] what's going on with hangouts [14:05:29] qchris_away poke me when you are ready [14:14:24] hi drdee [14:16:32] heya [14:16:39] yo ottomata [14:16:43] yoyoo [14:18:22] drdee: ping. qchris is ready. [14:18:29] yo [14:18:35] can you add me to the bat cave? [14:18:40] let's talk about your email [14:19:01] * qchris starts google machine. [14:21:43] drdee: You are added to the event already. Does joining not work? [14:22:12] i am in https://plus.google.com/hangouts/_/a059945337f918afac969b4c9ef0328ed22a17c2 [14:22:19] (CR) Milimetric: [C: 2 V: 2] "This makes sense, the sum(count) is a good improvement. I'm wondering how the 30 day limit thing will play out, but let's see :)" [analytics/geowiki] - https://gerrit.wikimedia.org/r/85626 (owner: QChris) [14:22:28] Ok. I'll come over. [15:14:15] brb [15:50:23] (CR) QChris: "> I'm wondering how the 30 day limit thing will play out, but let's see :)" [analytics/geowiki] - https://gerrit.wikimedia.org/r/85626 (owner: QChris) [16:03:53] whoa, drdee, qchris, i'm playing with jq to check up on the varnishkafka json stuff [16:03:55] its awesome [16:04:02] http://stedolan.github.io/jq/manual/ [16:04:14] * qchris looks [16:06:53] ottomata: jd looks neat :-) [16:08:14] oooo heya Snaps_ [16:08:21] just found a bug in the json outputter I think [16:08:40] this is a key:value pair that is in my output [16:08:40] "response_size":- [16:09:01] the default - when there is no varnish value should have quotes around it in JSON [16:09:19] hmmm, but ,"x_forwarded_for":"-" has it [16:09:20] hmm [16:09:26] OH! [16:09:35] because response size is a number [16:09:37] hm [16:10:35] we probably need a different default value if we are outputting json numbers, right Snaps_? [16:10:39] maybe 0, or −1? [16:10:51] or perhaps the - shoudl just be quoted: "-" [16:13:32] huhm [16:13:58] try the defaulter.. %{@response_size?0}r [16:14:03] (or whatever char it is) [16:14:08] ? [16:16:04] oh! [16:16:09] didn't know that existed, cool [16:16:23] also, btw, I just ran the same test I ran last week [16:16:26] 0 lost messages! :) [16:17:11] whoa! [16:17:16] yoyo [16:17:18] very good to hear. [16:17:25] what about reordering? [16:17:29] some messages out of order [16:17:32] but that is normal, right? [16:18:31] yeah, not much to do about that. [16:18:39] i could be counting wrong (i'm only looking for non contiguos seq numbers) but it looks like about 29 messages out of order [16:18:40] yeah, no worries [16:18:43] that is not a problem for us at all [16:18:49] out of order is normal i think, that's how kafka works [16:19:36] milimetric, around? [16:22:02] now for some esams tests! [16:22:15] ottomata: yah, both reordering and duplicates. [16:22:40] It'd be sweet if the kafka producer would add a sequence number per message so that could be sorted out on the consumer side with little effort. [16:22:49] Maybe in kafka 0.1000.0 [16:22:50] i didn't get any duplicates [16:22:57] well, i mean, we have that in varnish, right? [16:23:02] we put the sequence number in the message [16:23:05] so we can actually do that [16:23:08] yeah, but itd be more useful to have in the kafka transport itself [16:23:10] yeah [16:23:24] it'd have to be toppar sequence, rigth? [16:23:31] yep [16:30:03] so Snaps_ [16:30:06] %{@response_size!num?0}b [16:30:06] ? [16:35:48] yep [16:35:50] that should work [16:36:16] one could argue that the default should be 0 for numerics, as you said [16:37:25] i agree with Snaps_ [16:37:50] it's best to have a single data type in a column [16:38:18] mixing strings like '-' with numerics can only lead to more cleaning later on [16:38:23] aye [16:41:43] hm! so far, production from esams works exactly the same as production from eqiad! [16:42:51] oh yeah! [16:42:59] and thats transatlantic productions? [16:44:24] yeah [16:44:56] i'm testing with a single partition right now [16:45:09] going to test with 6 (i've got 6 disks for these brokers log.dir) [16:45:22] is this with live varnish logs? [16:45:27] i.e., thousands/s? [16:47:35] yes [16:51:27] Snaps_: i think i rejocied too soon, i think i was doing something wrong (the varnishkafka package wasn't fully installed on my esams node. libyajl didn't install since i'm using dpkg) [16:51:49] i'm unsure of what I just did that showed good stuff, but now i'm getting Failed to connect to broker at analytics1003.eqiad.wmnet:9092: Connection timed out [16:51:52] hang on... [16:55:37] drums please [16:58:59] Hey guys. Where do I find the hangout for the analytics standup? [16:59:10] halfak|WMF: /topic [16:59:24] Batcave? [16:59:31] affirmative [17:06:03] ottomata: ? :) [17:06:23] Snaps_: we are in scrum [17:06:44] :( [17:07:59] Snaps_: ha, i spoke to soon before, i must have been consuming from the wrong topic, must have been copy/paste error [17:08:11] there apparently are netowrking setup issue i need to figure out [17:08:13] ottomata: what's the link to the kafka ganglia stuff? [17:08:15] haven't actually produced anytihg else [17:08:40] tnegrin: : [17:08:46] http://ganglia.wikimedia.org/latest/?c=Analytics%20cluster%20eqiad&h=analytics1003.eqiad.wmnet&m=cpu_report&r=hour&s=descending&hc=4&mc=2 [17:08:51] gotta wait a bit [17:08:55] that page will take a while to load [17:19:19] * Snaps_ holds off the press release for a few more minutes [17:31:06] hey qchris, i'm blocked on the varnishkafka stuff for now, you sound really busy, but if you've got time i'd love to see where you are with hive/camus and work out production issues with ya [17:31:21] would like to see that as wel! [17:34:23] ottomata: Sure, but I do not really have much :-( [17:34:40] ottomata: And I mostly did this on my local Hadoop instance. [17:34:47] oh ok, can we make it work in labs together? [17:35:06] ottomata: Let me get something to eat and then I'll see if it works in the labs cluter as well. [17:37:04] hey ottomata -- are there any docs for the kafka metrics? it's a bit initiimidating [17:37:31] k [17:37:53] yeah there are a lot, I have trimmed them down a lot for production ones when they get puppetized, this is all of them on the test instances [17:38:35] tnegrin: http://kafka.apache.org/documentation.html#monitoring [17:38:55] great -- thanks [18:06:40] we're starting the research office hours in #wikimedia-office <-- join us if you're interested in talking data & research with community members [18:55:26] ottomata: Did you get hive to be nice in the meantime, or should we discuss the minimal things I know up to now? [18:55:40] i haven't been messing with hive since we last talked :) [18:55:47] Ok. [18:56:03] ottomata: Let me know if you want to discuss it. [18:56:08] ok yeahhh i do! [18:56:09] one sec [18:56:43] Sure. The problem is that I cannot ssh out from my google machine. So I probably cannot do a plain Hangout-Demo. [18:58:05] I could just send you some commands? [18:58:09] Or we do plain IRC? [18:58:14] hm sure, we can do screen [18:58:16] in labs [18:58:45] I haven't done screen in years :-) [18:58:49] haha [18:59:28] So basically I copied the stuff over to kraken-namenode-standby [18:59:48] in ~qchris/camus-data/prepared-data [18:59:56] I prepared some json files [19:00:12] Then I create the table in hive, and import them. [19:00:25] And I know how to query them. [19:00:33] ottomata i am in the batcave [19:01:14] ok qchris, doing 2 things at once, one second..:) [19:06:30] screen -r shared [19:06:34] screen -x shared [19:06:36] qchris: ^ [19:44:07] hey milimetric ! [19:44:33] milimetric: let's hangout when you feel like it [19:44:49] wanna show you some stuff, we're close to a new release of the survivor metric [19:44:50] hi average, ok, signing on [19:44:52] k [19:45:26] average send me a hangout invite or link if you are in a hangout [20:03:09] milimetric; backlog grooming [20:11:35] drdee: link? [20:11:39] batcave? [20:12:26] We have backlog grooming. Did not we have that on thursdays? [20:12:42] * qchris boots google machine again [20:14:37] drdee_, drdee: Where is the backlog grooming? [20:14:50] qchris: thursday is the story grooming [20:14:59] this is about what stories need to into analysis [20:15:09] so i can present them on thursday [20:15:11] ok. [20:15:41] Then I'll head off for some cake :-) [20:15:44] nice