[13:26:02] morning ottomata [13:26:18] how does your haunted city look like? [13:27:57] morning [13:28:05] not too bad, actually, at least in brooklyn [13:28:10] but subway tunnels are floooded [13:28:14] so its gonna be crazy for a while [13:28:20] people can't get to work, etc. [13:30:55] some places were hit very hard (just from the telly) [13:31:10] any good pics? [13:32:16] morning average_drifter [13:35:32] actually not so much, brooklyn was rather chill [13:35:33] at least the parts I was at [13:35:37] I checked out dumbo, [13:35:46] there was a car that had floated into the middle of the street [13:35:53] and you could tell that the streets were covered [13:36:01] the night before [13:36:01] but that's really it [13:55:19] drdee, wonder if you can help me with this sqoop import [13:55:22] its just dying and i'm not sure why [13:55:26] of coures [13:55:37] sqoop import --username root --password root --connect jdbc:mysql://localhost:3306/temp_data --split-by id --num-mappers 1 --query 'SELECT id, name FROM temp_table WHERE $CONDITIONS' --target-dir=/user/root/hive/temp3 [13:55:45] this is on my local vm [13:56:05] what is the log? [13:56:29] is 'id' the name of the primary key field? [13:57:03] yes [13:57:16] 12/10/31 13:54:31 INFO mapreduce.Job: Job job_1351610603237_0004 failed with state FAILED due to: Application application_1351610603237_0004 failed 1 times due to AM Container for appattempt_1351610603237_0004_000001 exited with exitCode: 1 due to: [13:57:16] .Failing this attempt.. Failing the application. [13:57:18] not much info there [13:57:26] just says it fails, not why [14:00:35] ughh [14:00:37] how do ipv6 local network ips look like ? [14:00:42] hey drdee [14:01:30] and the whole jdbc connection stuff is working properly? [14:01:40] average_drifter: ask ottomata :D [14:02:01] yeah, oh hey average_drifter! i need to review your thing [14:02:06] will do in a bit [14:02:24] ottomata: yeah, but before that [14:02:24] ottomata: how does an ipv6 local ip look like ? [14:02:24] the equivalent of 127.0.0.1 [14:02:28] or 192.168.0.xxx [14:02:32] or 192.168.xxx.xxx [14:02:38] in ipv6 [14:02:45] well, it depends, i think, there are long forms and short forms [14:03:16] loopback aka 127.0.0.1, could look like ::1 [14:03:45] which is also 0:0:0:0:0:0:0:1 [14:04:12] i'm not sure what our lan IPv6 addies are [14:05:47] moinmoin [14:10:31] hm, i think the jdbc stuff is working [14:10:40] actually, drdee, I don't really care about sqoop, i just want to test hue/hive [14:10:52] right :) [14:11:00] can I just copy a file from hdfs on kraken? [14:11:03] so why don't you copy some files from an01? [14:11:05] and create a hive schema or something? [14:11:15] yeah totally [14:11:46] ok, one of the /user/hive/warehouse/revision files? [14:16:50] ok, drdee, I copied one of the files, how do I make the revision schema? [14:17:03] go to hue [14:17:53] go to beeswax [14:17:55] go to tables [14:18:08] create a new table from file [14:20:36] NoSuchObjectException: NoSuchObjectException(message='default.revision table not found') [14:20:37] ? [14:21:03] ??? [14:21:15] ah my hue file browswer isn't working [14:21:15] hang on [14:23:37] hey drdee, reportcard stuff look good? I can update prod [14:24:03] let me quickly look [14:24:30] what's happening in china? [14:24:39] yeah, saw that [14:24:44] oppression? [14:24:46] :D [14:24:54] it's almost a 50% reduction [14:25:13] seems consistent though [14:25:13] like, a trend [14:25:20] a downwards trend that is [14:25:41] let's push it [14:27:28] k, live [15:52:43] ottomata, shall we beat hue? [15:53:10] i'm tring to beat it! [15:53:17] i have MRv1 running on my local vm [15:53:20] step by step... [15:53:25] ha [15:53:34] k [15:53:57] oh i did a pretty significant restructuring of the wmf-analytics/kraken repo [15:54:06] oh? [15:54:06] i am gonna wait with pushing it [15:54:10] howso? [15:54:20] basicaly folder reorganization [15:54:22] that's probably cool, it was pretty much just random stuff so far [15:54:23] cool [15:54:29] push it to a branch? [15:54:34] or whatever [15:54:48] I renamed 'bin' to 'bash' [15:54:56] as bin collides with eclipse as the default output folder [15:55:04] and it would delete all files in bin [15:55:11] very annoying behavior of eclipse [15:55:21] i renamed src to lang [15:55:37] and so lang has now different languages like 'avro', 'java', 'pig' [15:55:44] and the java part is completely mavenized [15:55:54] the java contains pig udf's [15:58:33] hmmm, ok [15:58:37] maybe bin can be 'scripts' rather than 'bash'? [15:59:06] ok [16:02:23] drdee! [16:02:24] it works with MRv1 :/ [16:02:28] aarrghghghg [16:02:42] great job for figuring it out [16:03:05] hm, myabe we should fix the bug with yarn :) [16:03:07] can you paste the log in a gist? [16:03:11] the log? [16:03:19] the hue output log [16:09:44] https://gist.github.com/3987944 [16:13:39] so, did we configure hue to work with the mysql metastore? [16:13:54] it seems that your vanilla installation uses the built-in derby based metastore [16:13:58] (which is totally fine) [16:14:08] but that's not the setup in kraken [16:15:00] we def did in kraken, my local setup is a wee hacky [16:16:34] IIRC, there are metastore config options in /etc/hue/hue.ini [16:19:37] yeah [16:19:44] pretty sure I set that up though…hmm but maybe not [16:19:56] actually, i betcah when I switched to MRv1 configs things got switched around [16:19:57] will check [16:21:49] wait, hue with the metastore? [16:22:11] no, we never set that up for kraken either [16:22:16] hive with mysql, yes [16:22:18] but not hue [16:23:00] i don't think we need to use mysql for hue [16:23:05] sqlite is fine [16:23:21] that is what kraken is doing too [16:23:28] sqlite for what? [16:23:36] hue database [16:23:38] that is the default [16:23:42] mmmmmmmmm [16:23:49] are you talking about hive? [16:23:50] we might as well use mysql [16:23:56] but hive is using mysql [16:24:09] and so if hue does not know that hive uses mysql it could lead to errors [16:24:16] (pure speculation) [16:24:34] i am not talking about hue's internal database [16:24:44] aye, ok, but hive is not configured in hue.ini [16:24:45] i am talking about hive's metastore and whether hue knows about this [16:24:50] right [16:24:52] ah, it should, no? [16:24:57] but…… [16:24:59] shoudln't it just submit jobs to hive? and hive deals with it [16:25:11] i don't know the exact details [16:25:14] of the architecture [16:25:25] hive is def using mysql on both my local and in kraken for metastore [16:25:39] go to line 282 in /etc/hue/hue.ini [16:26:36] hmmmm interesting [16:26:38] oh right [16:26:42] but that is the metastore service [16:26:43] not mysql [16:27:05] no…. we configured hive to use mysql as the metastore [16:27:05] yes [16:27:09] should 290 and 293 be enabled? [16:27:09] there are two hive daemons [16:27:11] hive-server2 [16:27:20] and hive-metastore [16:27:27] hive-metastore is configured to use mysql [16:27:33] right [16:27:37] these ports are for hue to talk to the hive-metastore service [16:27:38] not for mysql [16:27:48] and those are defaults, they are enabled [16:27:54] you can even see that in the hue logs [16:28:16] 12/10/31 09:01:30 INFO metastore.HiveMetaStore: 1: get_table : db=default tbl=revision [16:29:17] yes, but i've never seen that line (IIRC) when we were running the hue job on kraken [16:29:50] yeah, but its got to work, you can run simple non MR job hive queries in hue on kraken [16:29:57] select * from revision limit 10 works [16:30:26] if hue couldn't talk to hive server and/or hive metastore properly, it wouldn't be able to do that [16:32:18] mmmmmm wait another idea :) [16:32:32] i compared your log out with one from running on kraken [16:33:43] ottomata: you're not under that crane, right? http://www.theonion.com/articles/mom-just-called-to-make-sure-youre-not-under-that,30166/ [16:34:13] naw, i'm cool, this is me: [16:34:13] http://www.theonion.com/articles/misinformed-man-riding-out-storm-in-bathtub-filled,30151/ [16:41:38] this is going to be a fun few days. [16:41:47] internets were in Previous Roommate's Name [16:42:21] now i get to experience the joys of being put on hold by a local internet provider [16:57:12] "i firmly believe all politicians and policemen can be rehabilitated." mixtapes have the best names. [16:59:22] Staaaaaand up! [16:59:40] https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90 [17:00:06] still talking to robla….might be a few minutes late [17:28:25] milimetric: GO TO BED [17:28:47] TAKE THE DAY OFF [18:48:52] ok, drdee, i'm going to switch back to yarn on my local and see i I can figure this one out. Its gotta be something dummmbbbbbbb [18:49:01] :D [18:50:54] i was talking to drdee about this, but it might be of interest to others (esp analysts) [18:50:55] http://www.amazon.com/gp/product/0262517345/ref=as_li_ss_tl?tag=lessly0b-20 [18:51:05] it's the philosophical perspective on induction [18:51:20] a difficult question in the phil world [18:54:06] probably mostly of interest to erosen or DarTar :) [18:55:00] gonna disconnect internet for a few minutes..... [19:01:51] back [19:05:28] heya binasher, what's the scoop on pixel/event logging for kraken stuff? [19:06:17] no scoop [19:06:25] esp getting another path (other than /event.gif) set up in varnish [19:06:31] we don't want to pollute e3's datastream [19:08:22] every group should totes have their own pixel service! we'll have event{1..100}.gif [19:09:04] ... [19:34:11] is Sumana around ? [19:35:28] you check the staff channel? [19:38:38] dschoon: #wikimedia-staff ? [19:38:47] just a guess, dunno what the staff channel is [19:38:55] yes [19:38:56] that's right [19:39:41] yep, tried, says "you must be invited" [19:40:05] ah. [19:40:16] she's not online right now [19:40:23] probably due to the hurricane on the east coast [20:10:54] dschoon: the Jean Nicod lectures come from the lab where I did my PhD :) [20:11:10] hot. did you hang out with Gil? [20:11:16] I attended the whole series by Harman, not the most engaging speaker in earth but definitely clever guy (and the book is awesome for cognitive scientists/philosophers of science) [20:11:16] he's a pretty cool guy [20:11:21] Gilberto? [20:11:25] that's exactly what i was about to say :) [20:11:30] Gil Harman [20:11:30] :p [20:11:59] but then again, analytic philosophers are not known for being engaging [20:11:59] not that much, we had him around for one week in the lab [20:12:36] excepting Nozick [20:12:43] who was fucking funny [20:13:01] (http://en.wikipedia.org/wiki/Robert_Nozick) [20:13:02] it depends: if you do analytic philosophy of neotomism you probably enjoy TED-like audiences [20:13:27] brb [20:13:30] yeah, it's nice that things are getting more cross-discipinary [20:14:33] back [20:14:52] drdee: sorry to miss the scrum this morning, did it happen? [20:15:28] DarTar, yes it did, i spoke with aaron, jonathan and ryan about mysql - hive datatype mappings [20:15:29] it did. just the hive-mysql stuff was discussed [20:15:51] any notes/pointers or should I just look up the link that you shared? [20:16:44] I just wrapped up the biggest item on this week's todo list and I'm starting to decompress [20:16:59] nice, DarTar. congrats. that always feels good [20:17:19] look up the wiki page that i sent yesterday [20:17:27] k [20:38:29] ottomata, any luck with hue? [20:40:04] i've been chatting with event.gif people, came to some conclusions, writing email before I head out to move a piano [20:40:37] k [20:43:48] dschoon, btw my initial pom layout was inspired by the one in avro :) [20:44:08] woo [20:48:18] ok pom is (sort of) ready [20:48:29] i mean it compiles and it works [20:48:32] but man are there a lot of options [20:48:59] dschoon, final question about .classpath [20:49:13] so i can commit it but it will also have some lines that are specific to my dev machine [20:49:16] that's okay [20:49:16] ? [20:50:28] no, that is bad. [20:50:37] maven should generate the classpath for you. [20:50:41] you should never have a .classpath file [20:50:45] for eclipse? [20:52:46] * drdee is reading maven-dependency-plugin docs [20:52:58] correct. [20:53:06] if you haven't installed the maven plugin, you should do so. [20:53:16] (the mylyn connectors are optional) [20:54:35] i have [20:54:42] i didn't know that this was possible [20:55:05] ha [20:55:12] it's one of the major reasons why maven is great. [20:55:18] it has IDE integration for everything. [20:55:38] you can even go "Check out maven project from SCM" and it clones from git and sets everything up automatically [20:55:46] while you make a sandwich [20:58:37] okay so no .classpath just a working pom [20:58:51] and then you do clone the repo in eclipse using maven and it will be fine? [20:58:52] ja [20:58:54] gusyyya aaaahhhh [20:58:55] i have to run [20:59:17] piano moving time [20:59:17] does it float? [20:59:17] got to get uhaul before place closes [20:59:17] i hope so! [20:59:17] good luck! [20:59:17] hm, i need to find a ramp…. [20:59:25] i think it's "New Project > Import from SCM" and then you pick "Maven Project" in some random wizard, because Eclipse *loves* wizards. [20:59:25] can 4 boys lift a piano into a truck? [20:59:31] we will soon see... [20:59:31] laters! [20:59:35] laterz! [20:59:36] i hope you still have back muscles tomorrow! [20:59:49] dschoon, i am ready to push [21:00:50] do eet [21:01:46] done [21:02:06] k [21:02:06] getting lunch [21:02:07] will look at it afterward [21:02:13] ok [21:07:22] brb lunch [22:05:22] Happy halloween everybody! I am out!