[13:56:07] morning everyone [14:07:58] hello milimetric :) [14:08:10] I have done some changes overnight regarding the whitelisting of users to trigger tests [14:08:13] that "should" work [14:08:19] though I haven't tested on analytics repos [14:46:49] hey hashar [14:46:55] looo [14:47:01] i don't think i had a problem triggering tests [14:47:16] i just needed a way to get my dependencies installed [14:47:23] ahh [14:47:24] i got everything else working on travis-ci [14:47:25] https://travis-ci.org/wikimedia/limn [14:47:28] which project is that and what do you need ? [14:47:40] oh yeah limm [14:47:48] well so i basically need npm to run install and test [14:48:01] dschoon said there was a way to get a local npm install [14:48:10] like - download it and us it only in my folder [14:48:25] would that be ok policy wise? [14:49:46] I am not sure [14:49:58] the root issue is that we do not want to download untrusted code from a third party. [14:50:13] that being said, that does not solve your issue though :-] [14:50:37] the long term goal is to have our Jenkins builds isolated in vagrant virtual machines, so we would be able to download from a third party. [14:51:14] meanwhile, we want to provide trusted code which is either ship a copy of the npm module in limm or setup a different repo that host the deps. [14:51:42] morning guys [14:52:53] morning drdee [14:53:01] mornign! [14:53:03] i'm pondering what antoine's saying [14:53:13] i only got the last 5 sentences [14:53:37] so the mechanics of testing is done because it works on Travis CI [14:53:51] but one last hurdle to get it working on Jenkins is npm [14:53:55] about the html table link, i liked the idea of lazy loading, probably we would have to write it ourselves but could just that link as inspiration [14:54:00] because it downloads code from third parties [14:54:05] yeah npm remains a bitch [14:54:25] well so i was going to shelve this then until the next sprint [14:54:29] we really need to have a script that creates a debian package from a set of npm apps [14:54:36] so i changed the task from Jenkins integration to CI integration and marked it done [14:54:41] yea [14:54:42] ok [14:55:08] so David's focusing on the map thing [14:55:28] and I have one more bug left on limn and then I can do something cool [14:55:48] david and i were talking about what that cool thing might be [14:56:10] we have a bunch of ideas - it has to bring immediate value and be something WMF hasn't been able to do before [14:56:36] and requires some data crunching [14:56:52] so - i was hoping to write my first pig script today and translate it into hive as well [14:57:05] are you around to nudge me in the right direction? [14:59:59] the pig scripts we already have, try to get limn to read data from kraken [15:00:14] so then we could visualize the mobile page views in limn from kraken [15:00:24] that would be very cool IMHO [15:00:40] oh yea the proxy data source thing [15:00:41] ok [15:01:25] that would really kickass, also limn could maybe have a notion of 'live dataset' [15:01:32] milimetric: drdee: maybe you could talk about ops how to get npm modules deployed on WMF machine [15:01:47] we will probably end up with the same issue to handle when limm is deployed in production. [15:01:52] so in the yaml file you can specify how often the data is refreshed and that would trigger limn to reload the dataset [15:02:01] we are certainly not going to npm install there [15:02:13] hashar: we know what to do, create a debian package from npm apps [15:02:19] yea [15:02:21] that's all that ops wants [15:02:39] milimetric: you like the idea of live dataset? [15:02:42] this was david's last deploy code: http://git.less.ly/?p=kraken-ui.git;a=tree;f=fabfile;h=e1bc38af690b6b9809b0f459cb2d4917930d9de3;hb=HEAD [15:03:28] morning ottomata [15:03:30] you're right that we should do this sooner than later though [15:03:43] i am so bad in taking a day off [15:03:54] re 'live dataset', drdee, yeah, definitely, that would require a few changes on the graph so it doesn't have to refresh the page [15:03:57] :) [15:04:03] asana task? [15:04:04] :) [15:04:13] gotta pick up some ups stuff [15:04:24] i'll be lurking around [15:09:32] morrn! [15:11:32] heh, that's why dschoon's a friggin genius [15:11:39] one liner fix for the bug I was having [15:11:46] Trait.FG_LAYER [15:11:54] changes the zoom brush so it renders on top of everything [15:28:09] ping average_drifter [15:44:43] drdee: pong [15:45:38] hoookayyyy drdee, can I help you fix your oozie thing today? [15:45:52] i certainly hope so! [15:45:59] so upgrading did not fix it [15:46:43] the problem remains java.lang.ClassNotFoundException: dk.brics.automaton.RegExp [15:47:12] i suspect that this class should be provided by the pig.jar in hdfs://user/oozie/share/lib/ [15:47:43] so maybe there is a way to check whether one of the jars in hdfs://user/oozie/share/lib/ actually contains dk.brics.automaton.RegExp [15:48:03] if not, then that's definitely a bug, if yes then it's a class path issue [15:48:53] hm [15:52:01] can you grep contents of a jar file? [15:52:50] average_drifter: new patch set ready? [15:53:11] drdee: underway, please see asana for progress https://app.asana.com/0/1891117540465/2806598566114 [15:53:27] ty [15:54:15] "Options are digging through code, contacting a former Director of Mobile, or re-defining them so we can document them and after that we know what the all the numbers mean." [15:54:31] either go with 1) but if that takes too much time, go with 3) [15:54:44] ok [15:54:45] 2) there is no such thing, it's tomasz and has always been tomasz [15:55:15] if nobody knows what it means we might as well redefine it right away [15:55:15] drdee: I understood from Erik that it was Phil , I forgot the last name [15:55:17] drdee, yeah, will try that in a sec, reading some stuff [15:55:25] you can jar -tvf file.jar | grep [15:55:37] oh, he was Product Manager mobile, but he left [15:55:41] so don't think we can ask him [15:56:01] ok [15:58:20] ottomata, you can read this http://blog.cloudera.com/blog/2012/12/how-to-use-the-sharelib-in-apache-oozie/ [15:58:28] but i am pretty sure that we installed share lib correctly [15:58:54] and that we use it the right way in the oozie job [15:59:17] because if that part is not correctly installed you would get a java.lang.ClassNotFoundException: org.apache.pig.Main exception [15:59:29] but our bug is that pig actually get's successfully launched [15:59:38] but that one the pig internal functions is not working [16:02:49] drdee, what's a hdfs file that you are using for $INPUT on this? [16:02:53] with Gutenberg? [16:03:18] no, that was a simple test case to illustrate the problem on different clusters [16:03:31] the input is /wmf/raw/webrequest/webrequest-blog/ [16:03:33] ok [16:03:57] so all the scripts are in /user/diederik/blog/ [16:04:29] and this is the workflow: http://hue.analytics.wikimedia.org/oozie/edit_workflow/30 [16:09:23] drdee, i just ran your pig script in grunt and got it to work [16:09:30] yes :) [16:09:36] the pig script works in pig [16:09:38] ah ok [16:09:53] ok so I also grepped all of the sharelib jar files and didn't find that class [16:10:03] it's the launching from oozie that is the issue [16:10:40] mmmmmmmm [16:10:43] hold on [16:11:45] this was the bug that 4.1.2. fixed: https://issues.cloudera.org/browse/HUE-877?focusedCommentId=14983&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14983 [16:12:26] reply to my post: [16:12:28] "If you have the automaton.jar (or whatever jar has dk.brics.automaton.RegExp), you can try manually adding it to the Pig sharelib directory and that might resolve the issue. [16:12:29] " [16:12:33] see https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/02tpYiPmmWE [16:12:39] right i've been reading all those [16:12:55] so maybe search for automaton.jar and copy that to hdfs://user/oozie/share/lib [16:13:16] mayyybe that will work, but that seems like the wrong way to fix the problem, if the thing works in pig, right? [16:13:31] does grunt use a different pig jar than oozie? [16:13:56] don't know to be honest [16:14:19] well if that would fix it then we know there is a missing dependency and we could file a bug and get it properly fixed [16:14:19] ah there is a /usr/lib/pig/lib/automaton.jar [16:14:33] but not on hdfs? [16:14:38] righht [16:14:43] alllright! [16:14:45] that's it! [16:14:50] thanks!!!!!!!! [16:15:06] so i would definitely call this a bug :D [16:16:15] ok, so i'll put that in sharelib and can you try again? [16:17:32] ok yeah, try now [16:18:04] OH, hmm, should I reinstall sharelib if I upraded to cdh 4.1.2??? [16:18:10] the pig jar in sharelib is [16:18:11] /user/oozie/share/lib/pig/pig-0.10.0-cdh4.1.1.jar [16:18:52] the one that comes with 4.1.2 (not yet in hdfs) is pig-0.10.0-cdh4.1.2.jar [16:18:57] i'm going to reinstall sharelib real quick [16:19:02] AND then put automaton.jar in [16:19:59] aigh! [16:21:44] hokay [16:21:46] there, try now [16:28:14] launched..... [16:28:44] check http://oozie.analytics.wikimedia.org/oozie/ [16:28:47] still seems to be running [16:28:50] i gotta go now [16:28:56] probably back in a couple of hours [16:29:02] finalizing xmas shopping [16:29:03] ok cool [16:29:07] SUCCEEDED! [16:29:09] WOOOTO WOOO T [16:29:12] it worked [16:29:14] Cool [16:29:22] thanks so much ottomata and louisdang! [16:29:25] really col [16:29:30] talk laterz [16:29:37] See ya [16:31:14] ok cool i will update that cdh thread [17:58:58] https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90 [18:40:49] drdee, wanna chat today? All my meetings got cancelled [18:52:04] i think drdee is mostly off today [18:52:09] and for a few weeks I think [18:52:12] he's on baby duty [19:07:07] drdee is off until Jan 2nd [20:44:28] HELLO Everyone! [20:44:41] are we doing demo friday? [20:44:43] anyone interested? [20:45:33] erosen, ottomata, I've gotta catch a flight tonight, so maybe we can postpone 'till next Friday? [20:45:46] hellloooooo [20:45:55] yeah [20:45:58] awww but you had so many questions about my colors! [20:46:03] that's fine [20:46:07] i'll ask them ottomata [20:46:14] haha, ok [21:01:08] ottomata: I'll definitely ask those questions again sometime soon. [21:01:16] ok :) [21:01:23] where are you flying? [21:03:11] nashville tn [21:03:26] my brother claimed christmas this year [21:03:39] i'm super excited, my miracle berries just came fresh from FL [21:04:08] (http://en.wikipedia.org/wiki/Synsepalum_dulcificum) [21:06:48] yoyo [21:06:54] are we doing demo friday? [21:08:36] milimetric, ottomata, erosen ^^ [21:08:41] ottomata: did you still want to ganglia chat? [21:09:00] sure! [21:09:02] sorry [21:09:03] yeah yeah [21:09:04] i just signed in i guess, drdee: what is above you last comment? [21:09:27] : (http://en.wikipedia.org/wiki/Synsepalum_dulcificum) [21:10:00] drdee coming too? [21:10:05] on my way [21:46:45] oh, i mentioned this in an email [21:46:53] but i'm taking monday off [21:47:06] but i'll hop on for the 2 hour sync up meeting dsc scheduled [21:49:18] :) [22:05:20] YES! [22:05:20] YES! [22:05:21] got it [22:05:24] ldap groups work now [22:05:26] yayyyyyYYYAYYAYAYAYAY [22:07:37] drdee, this means that we can grant access to hdfs files and hue apps via labsconsole projects [22:07:53] WOOOOOOOOOOT [22:07:58] hue doesn't even really matter, the group permissions apply via hdfs cli too [22:08:25] awesome otto, you really kick ass! [22:08:27] so uhhh, next week i'll start working out proper permission setup for stuff in /wmf [22:09:45] niiiiiiiice [22:48:57] laters all, murray xmas!