[10:17:55] (CR) Nuria: "I think there are only two things needed to merge this changeset:" (4 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/111914 (owner: Milimetric) [15:23:29] (PS1) Ottomata: Adding symlink of lib/camus.jar to current version of camus [analytics/kraken] - https://gerrit.wikimedia.org/r/121386 [15:23:51] (PS2) Ottomata: Adding symlink of lib/camus.jar to current version of camus [analytics/kraken] - https://gerrit.wikimedia.org/r/121386 [15:45:42] (PS35) Nuria: [WIP] Run recurring reports using the scheduler [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric) [16:50:59] So where does the native snappy library live on your system, and where does hive/hadoop try to load it from? [16:51:37] hm, moment [16:52:21] I found this somewhere -Djava.library.path=/usr/lib/hadoop/lib/native [16:52:33] I guess this is how I tell hadoop where to get native libs [16:53:17] as for libsnappy, it seems to be living in /usr/lib [16:53:17] /usr/lib/libsnappy.a [16:53:18] /usr/lib/libsnappy.so [16:53:18] /usr/lib/libsnappy.so.1 [16:53:20] /usr/lib/libsnappy.so.1.1.4 [16:53:30] That's weher it is on analytics1010 [16:53:37] find /usr/lib/hadoop | grep snappy [16:53:38] on my machine [16:53:40] yields: [16:55:44] Seems my IRC client pasted things to the wrong channel. Sorry for the noise. [16:57:54] s'ok we shoudl talk in here [16:57:57] i was saying similar things in a PM [16:58:00] with average [16:58:40] Ok :-) [16:59:55] * qchris should not blame my IRC client. It was for sure me that got it wrong :-) [17:11:14] qchris, average, shoudl we remove kraken-dclass from kraken? [17:11:21] Yes! [17:11:31] I have patches more or less ready to do that, but! [17:11:41] Thing is, many pig scripts rely on dclass. [17:11:48] And we need to remove it from them as well. [17:11:57] So we need to clean them up beforehand. [17:13:04] That was the next thing I wanted to fix once we have a bit of not totally busy time. [17:13:13] But I am not sure when that will happen :-/ [17:14:41] ottomata: If you've got time to do it ... run wild on those pig scripts :-) [17:16:28] ok cool, that's fine [17:16:41] naw, i'm just trying to build, and was trying to minimize dependencies i have to add to archiva [17:50:16] (CR) Ottomata: [C: 2 V: 2] Adding symlink of lib/camus.jar to current version of camus [analytics/kraken] - https://gerrit.wikimedia.org/r/121386 (owner: Ottomata) [17:51:07] (PS1) Ottomata: Pointing deployment to archiva.wikimedia.org [analytics/kraken] - https://gerrit.wikimedia.org/r/121414 [17:52:46] (PS1) Ottomata: Adding kraken-toolbelt snapshot jar via git-fat [analytics/kraken] - https://gerrit.wikimedia.org/r/121415 [17:53:03] (CR) Ottomata: [C: 2 V: 2] Adding kraken-toolbelt snapshot jar via git-fat [analytics/kraken] - https://gerrit.wikimedia.org/r/121415 (owner: Ottomata) [17:53:53] (CR) Ottomata: [C: 2 V: 2] Pointing deployment to archiva.wikimedia.org [analytics/kraken] - https://gerrit.wikimedia.org/r/121414 (owner: Ottomata) [18:11:17] Hey milimetric, can you show me the costly survival query? [18:11:37] yes but in a meeting with Kevin [18:11:38] in a bit? [18:11:41] halfak: ^ [18:11:46] Sure [18:11:47] :) [18:12:40] qchris: how do I run toolbelt? [18:12:55] java -jar kraken-toolbelt-0.0.2-SNAPSHOT.jar org.wikimedia.analytics.kraken.toolbelt.Info [18:12:55] ? [18:12:56] Look at the readme :-) [18:13:00] no main manifest attribute, in kraken-toolbelt-0.0.2-SNAPSHOT.jar [18:13:05] yeahi thin i am just java noob [18:13:07] (CR) Csalvia: [WIP] Run recurring reports using the scheduler (7 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric) [18:13:12] Use classpath. [18:13:27] Let me find the demo script again ... [18:14:00] /home/qchris/toolbelt-demo.sh [18:14:06] ^ ottomata [18:14:12] (on analytics1010.wmf) [18:14:55] If you want to run outside of hadoop (directly using the java executable), use -cp instead of -jar [18:15:05] oh hadoop jar! [18:15:24] got it! [18:15:25] great [18:15:53] greaat [18:15:54] got it [18:15:55] coool [18:16:03] Yippie \o/ [18:16:43] so, i just ran a camus jar that was deployed via git deploy to import from kafka, and then i just ran toolbelt jar that was deployed via git-deploy to dump the contents of that file [18:16:45] woooo [18:16:53] i think we need to have some talks about reorganizing kraken repo soon [18:16:55] for stuff like this :) [18:16:57] and versioning, etc. [18:16:59] but soon@ [18:22:57] (CR) Milimetric: [WIP] Run recurring reports using the scheduler (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric) [18:25:14] anyone around to help with git? [18:28:27] what's up terrrydactyl? [18:29:08] ottomata: i tried to run a git commit --amend and got this: Merge branch 'master' of https://gerrit.wikimedia.org/r/p/analytics/wikimetrics [18:29:13] lost the change id [18:29:23] not sure where i went wrong [18:29:25] oh, so you probably did [18:29:28] git commit [18:29:29] ... [18:29:30] git pull [18:29:33] git commit --amend [18:29:37] hmm, probably! [18:29:42] sounds vaguely like something i did [18:29:59] yeah, you shoudl be really careful not to merge into your working branch if you have an outstanding change waiting in gerrit [18:30:06] is your change hanging out in gerrit somewhere? [18:30:30] https://gerrit.wikimedia.org/r/#/c/119343/ [18:31:13] ah, and it is on your master [18:31:14] hmm [18:31:23] ok, there are probabnly smarter ways to do what I am about to tell you [18:31:25] but I have a way [18:31:34] its called: BLAST THE STUFF, START AGAIN :p [18:31:39] hahaha [18:31:40] this is on your master branch, hmmmm [18:31:42] ok so [18:32:06] hmm, actually, can you manually reapply the stuff you just tried to —amend if you had that commit clean? [18:32:31] my code changes? hmm, there's quite a bit [18:32:49] btw, in the future i would avoid committing to your local master branch, its better to work in a local tracking branch that tracks origin/master [18:32:58] so, in the future, when you want to work on a feature, i'd do [18:33:05] git checkout -b feature_name origin/master [18:33:18] that way you can always pull into master safely [18:33:22] without fear of doing this to yourself [18:33:22] yeah, i did that with my last project and then promptly forgot to do it for this one [18:33:25] aye [18:33:27] yeah, cool [18:33:27] hmmm [18:33:45] wait, did you commit —amend the current changes into the merge commit? [18:33:48] or did you just cancel? [18:33:56] i cancelled it [18:34:25] deleted the message and aborted it [18:39:35] ok [18:39:37] sorry [18:39:48] so yeah, so you have staged changes then? [18:40:03] terrrydactyl: ^? [18:40:13] hey ottomata [18:40:19] no worries, I'm here with T, I'll help out [18:40:20] hey [18:40:23] ooook :) [18:40:34] thanks for helping so far [19:07:26] hey halfak [19:07:34] hey YuviPanda [19:07:37] halfak: got a minute or so to talk about schema design for the apps? :) [19:08:20] Sure. I haven't had a chance to critically look at the link you sent out though. :\ [19:08:24] (CR) Nuria: [WIP] Run recurring reports using the scheduler (5 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric) [19:10:57] halfak: we wanna do editing / login funnel, and 'article navigation' ones, I think. [19:11:31] Article navigation sounds dangerous. [19:13:09] halfak: I am trying to figure out what that means :) [19:14:04] We could log page load timings or simply log the fact that an article was presented. Does the app do caching? [19:14:14] halfak: sortof. [19:14:19] halfak: iOS does a lot more than Android [19:14:29] halfak: I think it's more about seeing how ToC is activated, etc [19:14:44] ToC? [19:15:01] halfak: Table of Contents [19:15:11] halfak: "Article navigation: Frequency of search and browse. When people arrive at a page what feature brought them there? How far to people typically browse on a page? Do people use the table of contents? (suggestion to track app sessions and session engagement patterns)" [19:15:26] halfak: for Article Navigation [19:16:37] YuviPanda, I'm going to have to get back to you on this. It's going to involve interpretation of our data retention guidelines. [19:16:42] halfak: ok. [19:16:53] halfak: https://trello.com/c/NsCra66X/1-event-logging-editing-signing-in-article-navigation [19:17:01] https://meta.wikimedia.org/wiki/Data_retention_guidelines [19:17:11] (anyone can comment) [19:17:16] I'll add that link there [19:17:31] I've got a lot on my plate. I'll have to get back to you tomorrow. [19:17:32] halfak: why this specifically? All this will just use regular EventLogging just like everything else, no? [19:17:34] halfak: ok! [19:17:40] halfak: sorry for being abrupt :) [19:17:41] err [19:17:44] We don't log page views in event logging. [19:17:46]

[19:17:51] halfak: right. [19:17:53] no worries. [19:18:06] closing for the nite ciaoooo [19:18:14] nuria: cya! [19:19:45] woops [19:20:53] halfak: heh. {{ec}} :) [19:21:07] yup. [19:21:13] halfak: ty! [20:07:43] (PS5) Terrrydactyl: Removing cohorts from database. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 [20:07:45] (PS1) Terrrydactyl: Change button from /upload to "Create Cohort" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/121448 [20:09:45] (CR) Milimetric: [C: 2] Change button from /upload to "Create Cohort" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/121448 (owner: Terrrydactyl) [20:09:56] (CR) Milimetric: [V: 2] Change button from /upload to "Create Cohort" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/121448 (owner: Terrrydactyl) [20:11:26] (Abandoned) Milimetric: Change button from /upload to "Create Cohort" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/121448 (owner: Terrrydactyl) [20:12:13] halfak: do you still have time for me? [20:12:18] sorry I made you wait so long [20:12:23] No worries. [20:12:29] Good timing. [20:12:35] ok, batcave? [20:12:46] (there's a new batcave: https://appear.in/wmf-analytics-batcave) [20:13:13] ottomata: I saw you added snapshot jars to kraken. ... How would I know where/how they got built if we add snapshots? [20:13:57] qchris, not sure what you mean [20:14:11] (btw, that was mainly just for testing, am all for rethinking how we add and deploy stuff now that it works!) [20:14:27] I am talking about https://gerrit.wikimedia.org/r/#/c/121415/ [20:14:42] How do I know which code is in there? [20:14:58] "there" = lib/kraken-toolbelt-0.0.2-SNAPSHOT.jar [20:15:01] (PS1) Terrrydactyl: Change button from /upload to "Create Cohort" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/121450 [20:15:23] So if a pig script would rely on lib/kraken-toolbelt-0.0.2-SNAPSHOT.jar [20:15:41] how could it know which commit the jar is based on (so it [20:15:51] knows what classes are in the jar) [20:16:28] (btw. I of course fine with testing. Just wanted to check whether I understand our approach) [20:23:42] qchris: thanks for responding. I was looking for that RfC but was too sleepy when I wrote the initial email [20:24:47] YuviPanda: Meh. I am just being a nit-picker :-P [20:24:51] It's fine to have the UA either way. [20:27:02] qchris, sorry, um [20:27:15] probably I should have added the file as a different name [20:27:21] maybe the actual snapshot date [20:28:18] ottomata: Well, we have to sort those things out one way or the other. [20:28:37] http://archiva.wikimedia.org/#artifact-details-download-content/org.wikimedia.analytics.kraken/kraken-toolbelt/0.0.2-SNAPSHOT [20:28:46] If we can afford it, a proper release would be great [20:28:47] the name of the file is actually irrelvant in this case [20:28:54] no snapshot jars in production! [20:28:55] yeah, once we sort that stuff out [20:28:59] with the system we worked out here [20:29:07] its only the sha of the jar that matters [20:29:14] the jar will be deployed based on the sha, not the name [20:29:18] Or a commit hash ... :-) [20:29:36] so, there has to be a sha mattching the sha1 sum of whatever the file happens to be in archiva [20:29:46] That's pretty cool. [20:29:55] you could do [20:30:03] But we need to get from the file to the commit hash of kraken one way or the other. [20:30:08] wget archiva.wikimedia.org/whatever.jar [20:30:13] mv whtaever.jar to thisotherthing.jar [20:30:17] git add thisotherthing.jar [20:30:18] git commit [20:30:23] and still [20:30:31] whatever.jar is what would be downloaded from archiva [20:30:34] since the sha1 will match [20:30:40] Ja. That's pretty cool. [20:30:43] I like that :-) [20:31:00] oh commit [20:31:00] hmmm [20:31:19] we might have to do that manually, if we want to [20:31:27] Ja. One way or the other, users of the jars need to know what classes/functionality they can expect in a jar. [20:31:29] like, if we add a snapshot, only add it by name of sha? [20:31:33] commit hashi mean [20:31:34] ? [20:31:56] i wonder if there is a mavne plugin that will deploy snapshots based on git commit hash [20:32:03] instead of timestamp [20:32:32] That sounds hacky still. [20:32:39] And "snapshot"-y. [20:32:49] A proper release would be even better. [20:33:07] Then we would have a well known and understand naming scheme to refer to. [20:33:09] true true [20:33:13] never adding snapshots is fine with me :) [20:33:20] But releasing is expensive (time-wise). [20:33:43] But I like the simple way to add git-fat files. [20:33:56] Looks super-simple. [20:33:59] *thumbs up* [20:34:32] We'll figure it out one way or the other. [20:36:38] yeah [20:45:25] milimetric, whenever you have some time, I want to steal it from you. :-) [20:58:55] (PS1) Milimetric: Fix revert rate prototype logic [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/121522 [21:00:21] (CR) Milimetric: [C: 2] Change button from /upload to "Create Cohort" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/121450 (owner: Terrrydactyl) [21:00:28] (CR) Jdlrobson: [C: 2] Fix flake8 errors [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/110196 (owner: Milimetric) [21:01:17] (PS1) Jdlrobson: Update the uploads schema definition [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/121525 [21:32:51] (PS6) Terrrydactyl: Removing cohorts from database. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/119343 [21:37:30] (PS1) Ottomata: Adding camus wrapper script and camus properties file for importing webrequest data [analytics/kraken] - https://gerrit.wikimedia.org/r/121531 [22:32:05] (CR) Milimetric: [WIP] Run recurring reports using the scheduler (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/112165 (owner: Milimetric) [22:32:51] (CR) Milimetric: [V: 2] Fix flake8 errors [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/110196 (owner: Milimetric) [22:33:00] (PS2) Milimetric: Update the uploads schema definition [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/121525 (owner: Jdlrobson)