[12:16:42] (CR) Mforns: Use Dygraphs in Vital Signs (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/214270 (https://phabricator.wikimedia.org/T96339) (owner: Milimetric) [12:23:05] (CR) Mforns: "'gulp js' gives me this:" [analytics/dashiki] - https://gerrit.wikimedia.org/r/214270 (https://phabricator.wikimedia.org/T96339) (owner: Milimetric) [13:30:27] (CR) Mforns: [C: -1] "When testing in the browser I noticed that:" [analytics/dashiki] - https://gerrit.wikimedia.org/r/214270 (https://phabricator.wikimedia.org/T96339) (owner: Milimetric) [13:50:58] (PS7) Mforns: Add stacked bars component to compare layout [analytics/dashiki] - https://gerrit.wikimedia.org/r/214036 (https://phabricator.wikimedia.org/T91123) [14:16:01] (PS1) Joal: Add host normalization udf and underlying functions [analytics/refinery/source] - https://gerrit.wikimedia.org/r/215628 [14:16:51] Hey milimetric [14:17:28] hey joal [14:17:47] I waited for you a bit, but I think you forgot our meeting :) [14:18:53] Or maybe not ? [14:18:58] omg [14:19:04] I'm so sorry [14:19:12] no prob it's ok :) [14:19:25] I had the hangout open, and did other stuff ;) [14:19:27] no, I just totally forgot [14:19:39] Well, that happens to me as well :) [14:19:45] my phone usually reminds me and I see now that it's not nearby [14:19:46] Nothing major so no issue [14:19:52] you wanna chat now? [14:19:59] If you want :) [14:20:03] sure [15:19:47] Analytics, VisualEditor, Wikimedia-Site-requests, Editing Department 2014/15 Q4 blockers, database: Backfill rctag data for VisualEditor from 2015-05-23–2015-05-28 - https://phabricator.wikimedia.org/T101270#1334021 (Jdforrester-WMF) NEW [15:44:05] Analytics-Kanban, Need-volunteer: Top Articles ad-hoc Report for Wikipedia Zero [5 pts] - https://phabricator.wikimedia.org/T99083#1334099 (ggellerman) a:Milimetric [15:44:24] Analytics-Cluster, Analytics-Kanban: Create current-definition/projectcounts [13 pts] {musk} - https://phabricator.wikimedia.org/T101118#1334101 (ggellerman) a:JAllemandou [15:44:54] Analytics-Cluster, Analytics-Kanban: Create new normalized uri_host field in refined webrequest table {hawk} [13 pts] - https://phabricator.wikimedia.org/T96044#1334103 (ggellerman) a:JAllemandou [15:53:28] Analytics-Cluster, Analytics-Kanban, Easy: Build component for Oozie jobs to sends e-mails {hawk} - https://phabricator.wikimedia.org/T88433#1334126 (kevinator) Open>declined We don't really need this at this point so closing the task. [15:58:38] (PS8) Madhuvishy: [WIP] Productionize app session metrics - Parse args using scopt - Move away from HiveContext to reading Parquet files directly - Change reports to run for last n days instead of daily or monthly (not sure if this is gonna work yet) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/212573 (https://phabricator.wikimedia.org/T97876) [15:58:51] joal: My latest changes are here ^ [15:59:35] madhuvishy: reading [16:00:53] mforns: I see a comment on a comment in my dygraphs patch, but you said you ran into an error [16:01:20] Things that we changes - 1. Reduce the depth of QTSemiGroup from 16 -> 6, 2. Get rid of the coalesce(100) in userSessions - andrew suggested that - "at the moment, reducing parallelism might reduce performance because more mem needed" [16:01:26] joal: ^ [16:01:43] k [16:02:14] Does it run now on 30 days ? [16:02:26] We also tried to add a persist(StorageLevel.MEMORY_ONLY_SER) to userSessions based on https://spark.apache.org/docs/latest/programming-guide.html#which-storage-level-to-choose [16:02:33] Nope. [16:02:50] ok [16:03:11] joal: this is the stacktrace now though - http://pastebin.com/hLHn6t2D [16:03:14] I am gonna try to see where it fails :) [16:03:17] the error is very different [16:03:40] you can see from first few lines that a whole stage finished [16:04:05] on how much data the stack trace ? [16:04:34] joal: i tried first with the persist. and then this happened - i tried without, but that dint work too. [16:04:37] 30 days [16:04:42] ok [16:05:04] joal: this is what i was running - spark-submit --master yarn --driver-memory 1500M --num-executors=40 --executor-cores=1 --executor-memory=2g --class org.wikimedia.analytics.refinery.job.AppSessionMetrics --verbose /home/madhuvishy/workplace/refinery-source/source/refinery-job/target/refinery-job-0.0.12-SNAPSHOT.jar -o /user/madhuvishy/tmp/ -y 2015 -m 5 [16:05:05] -d 26 -n 30 [16:05:10] I need to take care of Lino a bit, will be back in an hour or so, and try to help :) [16:05:24] joal: okay great. i'll commute to office now :) [16:05:29] cool :) [16:05:35] later ! [16:14:24] Analytics-Kanban, Labs: LabsDB problems negatively affect analytics tools like Wikimetrics, Vital Signs, Quarry, etc. {mole} - https://phabricator.wikimedia.org/T76075#1334179 (kevinator) Open>declined a:kevinator I'm closing this task because it is very broad and there are no clear next steps to... [16:52:59] (PS8) Mforns: Add stacked bars component to compare layout [analytics/dashiki] - https://gerrit.wikimedia.org/r/214036 (https://phabricator.wikimedia.org/T91123) [16:53:48] (CR) Mforns: Add stacked bars component to compare layout (23 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/214036 (https://phabricator.wikimedia.org/T91123) (owner: Mforns) [17:01:25] madhuvishy: any change ? [17:01:57] milimetric: i'm going to invite you to a meeting. would you be available 1435-1525 pacific time tomorrow? [17:02:30] dr0ptp4kt: I'll have to leave about 5 minutes early, but yes [17:03:57] milimetric: thx [17:04:33] milimetric: i made it 5 minutes shorter [17:16:56] joal: nope. it still fails at line 61. [17:18:06] Quick idea madhuvishy : Cna you check the number of tasks per job step ? [17:18:10] My guess is 200 [17:18:20] Just before the failure [17:19:17] hm, wrong from the stack trace you sent [17:23:15] joal: yeah, more like 46k? [17:23:21] yup [17:23:43] looks like it's far too much given the data size we expect ... [17:23:54] But still, better too much than not enough [17:24:40] yup [17:24:59] it manages to run one whole time [17:25:04] then fails [17:26:16] when you say one whole time, you mean a full step ? [17:26:23] joal: yeah [17:27:13] I am gonna try to run that in a shell, see if I can get more [17:27:33] joal: okay [17:46:42] joal, yt? [17:58:17] mforns: I am now :) [17:58:28] wasup ? [18:04:20] joal, hi! [18:04:24] :) [18:05:03] joal, can you point me to the project that contains the code that needs modification for https://phabricator.wikimedia.org/T99932 ? [18:05:26] joal, is it analytics-refinery_source? [18:05:33] Yessir P [18:05:42] ok [18:06:34] thanks! I'm gonna have a look [18:06:39] No prob [18:06:54] There are two places to look for: one in refinery-core package [18:07:03] The other in refinery-hive package [18:07:41] I guess you should go refinery-core, and hive should come by itself [18:07:45] mforns: --^ [18:08:15] joal, I see, looking at UAParser [18:08:21] great :) [18:08:30] thanks! [18:09:59] joal: did you try running the job? [18:10:12] no didn't yet [18:21:52] mforns: you around? [18:23:09] my IRC is really messed up, random messages keep appearing and disappearing from the chat rooms I'm in [18:23:29] I'm trying to reach mforns, if he's pinging me and I'm not seeing it, please email me :) [18:23:43] milimetric, hi! [18:23:55] ah! good [18:26:00] joal: I want to ask before I keep looking at scala docs: [18:26:12] is there a way to do a "partition" like in SQL? [18:26:32] because I'd want to basically partition over the projects and count the articles that way [18:55:02] (PS6) Milimetric: Use Dygraphs in Vital Signs [analytics/dashiki] - https://gerrit.wikimedia.org/r/214270 (https://phabricator.wikimedia.org/T96339) [18:56:30] (CR) Milimetric: "Addressed the problems with the build and race condition. The layout problem is a style issue, I'd love to get someone who's really good " (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/214270 (https://phabricator.wikimedia.org/T96339) (owner: Milimetric) [18:58:46] milimetric, there's still the lint complaint [18:59:10] did I not save? I changed it... [19:00:16] Analytics-Kanban: Stand up piwik in a permanent and privacy-sensitive way - https://phabricator.wikimedia.org/T98058#1335009 (Milimetric) From Michelle and Stephen (legal): "Generally, we think piwik should be ok if we configure it such that we are not collecting more than what's permissible under the main W... [19:00:56] milimetric, wait [19:01:05] I moved the ?: https://gerrit.wikimedia.org/r/#/c/214270/6/src/components/compare-layout/compare-layout.js [19:01:16] does that still throw a lint error for you? [19:01:28] if so, what version of jshint do you have, maybe i need to upgrade [19:01:40] milimetric, sorry I had the changes we applied and the pull did not work, so I was linting the old patch, sorry [19:01:49] gotcha [19:02:24] Analytics-Kanban: Stand up piwik in a permanent and privacy-sensitive way - https://phabricator.wikimedia.org/T98058#1335014 (yuvipanda) Way this should go should be: 1. Setup a piwik instance by hand, tweak the privacy knobs and what not to see how close to what we want to get it to we can get it to 2. If 1... [19:04:34] (CR) Mforns: [C: 2 V: 2] "LGTM" [analytics/dashiki] - https://gerrit.wikimedia.org/r/214270 (https://phabricator.wikimedia.org/T96339) (owner: Milimetric) [19:05:28] milimetric, merged it! and I'll sign off for today [19:05:40] see you tomorrow folks! [19:06:05] nite [19:06:09] thx! [19:10:42] milimetric: Hi [19:10:50] Was eating :) [19:11:15] Quick answer is no, there is no "parition" as in SQL 98 [19:11:32] But there are tricky ways of doing the same thing :) [19:11:53] It's late for me tonight, so I'll explain tomorrow if it's ok [19:12:28] madhuvishy: I have tried running some, but didn't manage to find either a solution or a root cause for our issue [19:12:38] Will continue to investigate tomorrow [19:12:50] joal: alright. i guess i'm on the same boat too [19:13:11] good night :) [19:13:27] Thanks madhuvishy :) [19:14:59] milimetric: still IRC issues I guess ;) [19:15:03] Will talk tomorrow :) [19:15:10] Have a good night folks ! [20:23:37] thanks, joseph, I could see a few ways to do it but it seemed very inefficient. Basically, get a data frame with the counts and join two copies of that aggregation together [20:23:42] but we'll talk tomorrow [20:33:16] "Failed to read artifact descriptor for org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.4.0: Could not transfer artifact org.apache.hadoop:hadoop-common:pom:2.6.0-cdh5.4.0 from/to wmf-mirrored (https://archiva.wikimedia.org/repository/mirrored/): peer not authenticated " can someone debug this? I hate you, maven. [20:33:22] *help me debug this [20:34:01] hm... [20:34:10] ottomata would probably know, but he's in transit, Ironholds [20:35:00] darnit :(. [20:35:46] Ironholds: I dont know much but where is this error showing up? [20:37:43] madhuvishy, when building refinery-source, basically [20:37:47] (well, when building before testing) [20:38:56] Ironholds: aah. are you doing something like mvn clean package? [20:41:07] Ironholds: I'd try to get rid of the folders in ~/.m2/repository and try again [20:42:10] madhuvishy, cool! Will try :) [20:42:55] Ironholds: also which version of maven do you have. i looked a little on Stack Overflow, and people claim this is something that was fixed in 3.0.5 [20:43:04] I'll check! [20:43:13] 3.0.5 :/ [20:43:34] Ironholds: hmmm. [20:44:38] i have 3.3.3 and can build refinery-source. try bumping may be. sorry I'm not being super helpful. [20:45:22] I'll do my best and see what happens :). Thanks! [20:45:35] Ironholds: :) [20:51:55] madhuvishy, this is apparently the default for ubuntu 14.04 and I've built under it before. Ah well; I'll poke 'drew when he's back [20:51:57] thanks for your help! [20:52:28] Ironholds: aah. alright! [20:52:45] also, everyone should be calling ottomata Mistah Otto [20:53:19] he should be in SF now. will call him that in office tomorrow [20:56:15] cool! [20:57:19] Ironholds: what is the etymology of the name :) [21:21:17] milimetric: can you think of any reason why the following query doesn't seem to go into mapping (at least the text output doesn't suggest as much) ? https://gist.github.com/anonymous/e4ce1b97df9579dbc0cb [21:21:34] one sec dr0ptp4kt, finishing up a meeting [21:21:42] milimetric: thx [21:22:38] (btw, i did a month > 3 earlier, but tried the ORs on the month in case it was not rifling through partitions correctly for some reason) [21:29:13] dr0ptp4kt: you mean it's just sitting there stalled? [21:29:50] what's the text output say? [21:30:14] (month > 3 should work just fine here, btw [21:30:16] ) [21:31:26] milimetric: initiating query, then gotta step away. will copy and paste the output once it stalls out again (unless i get lucky and it starts showing it's mapping) [21:32:01] dr0ptp4kt: it's possible it's just competing for resources with a lot of stuff [21:32:07] dr0ptp4kt: has happened for me before [21:32:07] that query would go over a *lot* of data [21:32:19] like... an expletive ton :) [21:32:41] yeah, i'd try with a day to make sure the queries alright, and then submit it and wait for over a day [21:32:59] agreed with madhu [22:23:37] milimetric: madhuvishy isolating it to one day works fine. another approach i've taken in the past is to check the same hour of every day. but in this case knowing the full day sums is going to be important due to timezones and how apps users may be distributed. [22:23:40] milimetric: do you have a second to look at my dashiki problems? [22:24:32] milimetric: madhuvishy it seemed at least last week that table sampling was also still resulting in full file scans, so no luck there [22:24:49] (for the query finishing faster) [22:26:48] dr0ptp4kt: aah. you should consider talking to joal|night tomorrow - he might know how to get better performance/get through it faster. [22:41:32] madhuvishy: thx [22:43:32] madhuvishy: for now i'm going to just try getting the stats on hour 1 - that seems to at least be running the mapper :) [22:46:43] dr0ptp4kt: cool. also, not sure if you already tried this - but https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Sampling on table sampling, suggests to sample by the same columns your data is clustered by [22:46:58] dr0ptp4kt: https://github.com/wikimedia/analytics-refinery/blob/master/hive/webrequest/create_webrequest_table.hql clusters by hostname, sequence [22:48:12] dr0ptp4kt: so may be doing TABLESAMPLE(BUCKET 3 OUT OF 64 ON hostname, sequence) will not look through all the data. i haven't tried this though [23:20:12] madhuvishy / dr0ptp4kt: I know second hand that TABLESAMPLE works and speeds up queries a lot [23:20:38] so if that's how you're sampling, Adam, then there might be some other problem. [23:20:49] kevinator: yeah, let's figure out why dashiki's not working [23:20:53] I'll be in the batcave [23:41:26] (PS1) Milimetric: Fix bad moment and twix version combination [analytics/dashiki] - https://gerrit.wikimedia.org/r/215823 [23:48:55] (PS2) Milimetric: Fix bad moment and twix version combination [analytics/dashiki] - https://gerrit.wikimedia.org/r/215823 [23:50:47] (CR) Milimetric: [C: 2 V: 2] "self-merging because this is an annoying version and hard to debug" [analytics/dashiki] - https://gerrit.wikimedia.org/r/215823 (owner: Milimetric)