[16:58:50] ok, thanks [17:34:50] nuria: http://twitter.github.io/algebird/index.html#com.twitter.algebird.QTree$ [17:35:06] With an interesting link to academic paper :) [17:35:40] joal: Thank you! that is reading homework for today after: http://www.eecs.berkeley.edu/Pubs/TechRpts/2011/EECS-2011-82.pdf [17:37:00] Thx nuria :) [17:37:50] joal: i have not used scla afor yeras now and i look at my code and i am like : ...did i wrote that? i have no recollection. [17:38:10] Yeah, that's the thing with functional programming :) [17:39:00] Stronger expressevitiy, less code so less bug, but sometime not much clarity :) [17:56:26] test [17:56:34] joal: you there? [17:59:51] ottomata: after solving issues with java vs there is still: [17:59:56] https://www.irccloud.com/pastebin/CbpLUz2B [18:00:08] kevinator: yup [18:01:30] (CR) Joal: "Coding style: leading or trailing dot when chaining methods ... Any reference?" (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/199935 (https://phabricator.wikimedia.org/T86535) (owner: Mforns) [18:02:10] ah! [18:02:11] weird! [18:02:13] where is that coming from! [18:02:42] hm [18:03:06] joal: can we talk briefly in the batcave? [18:03:13] yup, joining [18:03:33] ha, happening to me after I blew up my m2 cache nuria [18:03:37] i thin i can fix hang on.. [18:03:57] ottomata: ya, that is mvn issue, not code [18:10:44] ok nuria, try now [18:10:48] blast javaweh [18:10:50] from your m2 [18:10:52] and try again [18:11:39] ottomata: k [18:13:44] ottomata: good for that one, onto next one, what version of scala will we be using? [18:14:34] ottomata: I think i installed 2.11 on homebrew but lookslike we want to be on 2.10? [18:16:28] i chose 2.10 because that is the version that the spark that comes with cdh uses [18:16:43] i *think*... [18:19:15] nuria, ja [18:19:15] http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_vd_cdh5_maven_repo.html#concept_prw_wd2_yq_unique_2 [18:19:31] org.apache.spark [18:19:31] spark-core_2.10 [18:19:31] 1.2.0-cdh5.3.1 [18:20:01] ottomata: ok, changing that on my setup [18:47:53] ottomata: check this one [18:47:56] https://www.irccloud.com/pastebin/QTzK5H6I [18:48:39] ottomata: i think we need to go with 2.10.4 [18:49:08] i saw that, ja [18:49:13] ottomata: as 2.10.0 is not even in the download list (it might exists on mvn though) [18:49:14] http://www.scala-lang.org/download/all.html [18:49:19] but, all those others require it [18:49:31] ottomata: can we make it 2.10.*? [18:49:33] woudln't we have the same problme if we switched to 2.10.4, only the other way? [18:49:35] sure...>? [18:49:37] can we? [18:50:11] no, we would not, let me remember how did you that in mvn [18:54:06] (CR) Mforns: "The reason I used trailing dots when chaining is that: in the spark-shell the value definitions with no trailing dots are interpreted as s" (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/199935 (https://phabricator.wikimedia.org/T86535) (owner: Mforns) [18:57:49] ottomata: spark is 2.10 so i think we can do: [18:57:52] See y'all tomorrow ! [18:58:22] ciao joal|night [18:58:28] nighters [18:58:33] ottomata: specify major.minor [18:58:37] [18:58:37] org.scala-lang [18:58:37] scala-library [18:58:37] 2.10 [18:58:37] [18:59:05] but pack 2.10.4 in our archiva as such [18:59:25] goodnight [18:59:38] ? [18:59:39] ottomata: makes sense? [18:59:47] ottomata: ahem no... wait [19:00:10] ottomata: so we specify the dependency as this: [19:00:26] https://www.irccloud.com/pastebin/NXvkqQpg [19:00:48] so, spark-assembly.jar, which comes with cdh, includes scala 2.10 in it [19:00:54] And in archiva under that dependency we have packed version 2.10.4 [19:01:16] ottomata: but what version of 2.10? [19:01:50] ottomata: cause if we use "spark catalyst:" spark-catalyst_2.10:1.2.0-cdh5.3.1 requires scala version: 2.10.4 [19:01:52] nuria, my q is: why? to avoid the maven warning? [19:02:21] ottomata: well at runtime it will not work with both versions [19:02:22] your suggestion is to trick maven to avoid the warning, right? [19:02:28] i have run it! :) [19:03:38] ottomata: until we use the catalist features ( do not even know what that is...) [19:04:42] spark sql query optimizer (just googled it :) ) [19:04:49] ottomata: jaja meto [19:04:51] nuria, i think it will probably work, no? [19:04:53] me too [19:04:56] it is a minor version change [19:05:06] ottomata: at runtime? i doubt it [19:05:41] i have run it at runtime [19:05:42] ? [19:05:52] a minor api version shouldn't change the scala api at all [19:06:03] i suspect this is an oversight on cloudera's part [19:06:17] they even say to use 2.10 for that [19:06:17] org.apache.spark [19:06:17] spark-catalyst_2.10 [19:06:18] 1.2.0-cdh5.3.1 [19:06:22] http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_vd_cdh5_maven_repo.html#concept_prw_wd2_yq_unique_2 [19:07:04] well 2.10 could be either but that's fine, you want to leave it as is, correct? [19:08:23] ja, i would prefer to leave it than to try to trick maven [19:08:28] either way there will be a version mismatch, no? [19:08:55] ottomata: no, if we "move" to 2.10.4 but we do not have to do it now, it is not that important [19:24:06] ottomata: so - when you run mvn compile- as things aare for you right now it compiles just fine [19:25:03] yes [19:25:57] ottomata: ok, works for me too [20:42:16] (PS5) Ottomata: [WIP] Add Apps session metrics job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/199935 (https://phabricator.wikimedia.org/T86535) (owner: Mforns) [20:49:03] (PS1) Ottomata: Add oozie util workflow to launch spark jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) [20:51:43] (PS2) Ottomata: Add oozie util workflow to launch spark jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) [20:52:37] man that works! [20:52:42] but that was annoying to figure out! :p [20:56:04] ottomata: looking [21:06:25] (CR) Nuria: "And we have tested this works by running some dummy job right?" (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) (owner: Ottomata) [21:08:00] nuria, yup. [21:08:07] i used marcell's [21:08:23] oozie job -run -config ./oozie/util/spark/submit/workflow.properties -Dspark_app_file=hdfs://analytics-hadoop/user/otto/tmp/refinery-job-0.0.10-SNAPSHOT.jar -Dspark_main_class=org.wikimedia.analytics.refinery.job.AppSessionMetrics -Doozie_directory=/user/otto/oozie -Dinput=/wmf/data/wmf/webrequest/webrequest_source=mobile/year=2015/month=3/day=20/hour=0/000000_0 -Doutput=/tmp/foo8 -Dspark_num_executors=4 -Dspark_executor_cores=2 [21:10:35] (CR) Ottomata: "Yes, I used AppSessionMetrics WIP:" (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/201009 (https://phabricator.wikimedia.org/T94596) (owner: Ottomata) [21:12:00] laters [21:12:01] ! [21:12:16] ottomata:ciao, let's talk about this tomorrow