[14:22:48] morningggggg [14:31:34] average_drifter, you around? [15:18:22] morning ottomata [15:19:37] morning [15:21:49] read your server roles email in more detail last night. Sounds to me like if we can keep the vulnerable pieces up and running (like NamedNode) then we have enough storage to juggle the entire firehose as long as we delete un-analyzed data after a week [15:23:04] We can do more precise calculations and have a "offload it or lose it" policy for processed data after like 5 months or however the math works out [15:24:02] and then we can be very clear about how much we could extend that "offload it or lose it" time window if we had more hardware. [15:31:18] yeah, sounds good to me [15:31:28] we can also buy way more space if we set replication factor to 2 instead of 3 [15:31:37] 100TB vs 66 TB [15:35:01] yea, that sounds good but performance would suffer [15:35:13] right? or not how Hadoop works [15:36:44] performance probably not [15:36:45] just reliability [15:36:57] i think we'd probably be ok though [15:37:17] but, if we lost 2 servers at once, that happened to house the same data [15:37:19] we'd lose dat [15:37:20] data [15:37:24] 3 at once is much less likely [16:39:30] hey all, i'm going to shutdown hadoop in a bit in an attempt to move the namenode [16:39:35] just a warning, probably 20 or 30 minutes [17:02:09] good to know [17:15:05] hmm, well, actually, I am decomissioning an02-an10 right now [17:15:08] so that will take a while I thikn [17:15:20] its gotta copy all the data we've been putting on them for the last couple of months onto the new dells [17:15:28] cool [17:15:33] i am messing around in pig local mode anyway [17:15:42] do you think they will be back up today [17:15:42] ? [17:16:13] yeah [17:16:20] i mean, they should be useable right now, I guess! [17:16:23] hadoop is still online [17:16:27] its just doing a buncha copying all over the splace [17:16:29] place [17:16:35] not sure what happens if you need data that is being all moved around [17:16:40] i guess hadoop has lock mechinisms, i unno! [17:18:12] cool [17:44:09] ottomata: do you know anything about the provenance of the kraken.jar file? [17:44:30] i don't see anything in the maven pom file to generate one [17:44:39] but I can't believe it was done manually [17:45:26] oof, i have no idea [17:45:29] haven't messed with maven at all eyt [17:46:28] cool [17:46:36] i'll figure it out [17:56:20] https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90 [18:21:01] sometimes i wonder if i'd get to work faster if i'd walk [18:21:03] it's only 3.5 miles. [18:21:21] how can this possibly have a variance of 100% against the mean? [18:27:43] bicycle. fastest way to get around a city including driving [18:27:56] i am coming around to this view. [18:42:51] skateboarding ... [18:43:00] that way you can bus when it is on time [18:43:09] dschoon: quick maven question [18:43:15] sup [18:43:24] what is your goto plugin for creating a jar-with-dependencies [18:43:38] isn't that built in? [18:43:42] oh. [18:43:45] no, it's maven-assembly [18:43:45] looking around it seems like you can use maven-jar-plugin [18:43:49] or yeah [18:43:50] that one [18:43:57] yeah, assembly is the answer. [18:43:58] that is what I am using and it works [18:44:01] so I'll leave it [18:44:08] cool [18:52:33] also, you're right it does default to creating a jar when you tell it it package [18:52:47] yeah, in target [18:53:00] yeah [18:53:12] i just looked through the pom file and didn't see anything [18:53:23] or in the target dir of my repo [20:15:59] average_drifter hi? [20:57:26] it really makes me sad there's no /usr/bin/pet [20:57:33] i mean, we have /bin/touch [20:57:36] why can't we pet things? [20:58:01] or /usr/bin/pet —heavy [20:58:02] ? [21:01:13] no no. we're more modern and sophisticated than that. [21:01:21] we use double-dash flags. [21:05:14] i did too, but adium translates my double dashes to long em [21:05:20] heh. [21:05:35] dschoon, dunno if you will like this, but: [21:05:38] http://ganglia.wikimedia.org/latest/graph_all_periods.php?title=Kafka+Broker+BytesIn&vl=&x=&n=0&hreg%5B%5D=analytics102(1%7C2)&mreg%5B%5D=kafka_server_BrokerTopicStat.BytesIn>ype=stack&glegend=show&aggregate=1 [21:05:46] :D [21:06:00] will check it out in a sec [21:29:54] ottomata: ping :-) [21:30:10] ottomata: are you able to get your libraries to build on jenkins ? [21:30:21] https://gerrit.wikimedia.org/r/#/c/33466 [21:30:31] bwer? [21:30:34] i have nevah tried [21:30:40] oh! [21:30:41] with that stuff [21:30:42] ahh maybe that was Stefan [21:30:43] yes [21:30:51] yeah he was trying, but I helped him with this one [21:30:53] and it worked, yes [21:30:58] after I did this [21:31:03] and I did not know you had merge rights on puppet [21:31:08] yup! [21:31:09] :) [21:31:29] I need some more social engineering to get one myself \O/ [21:31:45] merge rights? :) [21:31:51] so I guess whenever Stefan need anything in puppet he will be able to ask you [21:31:55] heh, ja [21:31:55] that is nice, less troubles for me [21:32:05] erosen, dschoon, i'm getting ready to move the namenode, hadoop will be down for a few minutes [21:32:11] word [21:32:11] hot [21:32:12] thanks ottomata :) [21:32:14] thanks [21:41:10] guys, i'm taking that back [21:41:14] its almost quitting time for the weekend for me [21:41:20] i don't want to do this right now [21:41:24] i could def get the namenode moved [21:41:34] but an01 is hosting a buncha stuff that I betcha will break if I move the namenode [21:41:38] its possible it would be fine [21:41:46] but i'd rather not have to spend 2 more hours here fixing it on friday eve [21:41:47] so! [21:41:48] monday! [21:41:51] its all ready though [21:41:59] i've decommissioned the cisco hadoop datanodes [21:42:09] so, those are now available for use as storm/etl [22:42:21] Can anyone help me figure out what the element is in the new xml wiki dumps? It appears inside .