[01:31:22] <grrrit-wm>	 (PS4) Terrrydactyl: [WIP] Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091
[07:34:40] <grrrit-wm>	 (CR) Gilles: "This can be merged now, FYI. I've made sure that the new tables for the updated schema versions exist already." [analytics/multimedia] - https://gerrit.wikimedia.org/r/134065 (owner: Gilles)
[08:57:24] <grrrit-wm>	 (CR) Nuria: Refactor cohort methods into service (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134480 (owner: Milimetric)
[09:21:29] <grrrit-wm>	 (CR) Nuria: "Please add pertinent tests and handling of empty tags. Validation of ui input needs to happen at controller level (at least)." (4 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[10:55:50] <grrrit-wm>	 (CR) Nuria: [WIP] Add cohort class hierarchy, refactor CohortService (4 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743 (owner: Milimetric)
[13:31:32] <ottomata>	 qchris: https://gerrit.wikimedia.org/r/#/c/133425/
[13:31:33] <ottomata>	 better?
[13:31:43] <qchris>	 Just reviewing your commits.
[13:32:12] <qchris>	 We're having the same nits as with kraken-deploy back then.
[13:32:24] <qchris>	 But I'll merge and provide follow-up commits.
[13:36:23] <grrrit-wm>	 (PS1) QChris: Limit git-fat to artifacts subdirectory [analytics/refinery] - https://gerrit.wikimedia.org/r/134818
[13:36:29] <grrrit-wm>	 (PS1) QChris: Whitespace cleanup for gitfat configuration [analytics/refinery] - https://gerrit.wikimedia.org/r/134819
[13:37:27] <ottomata>	 :)
[13:37:34] <ottomata>	 ah i forgot about that artifacts/ limitation
[13:38:07] <grrrit-wm>	 (CR) Ottomata: [C: 2] Limit git-fat to artifacts subdirectory [analytics/refinery] - https://gerrit.wikimedia.org/r/134818 (owner: QChris)
[13:38:12] <grrrit-wm>	 (CR) Ottomata: [C: 2] Whitespace cleanup for gitfat configuration [analytics/refinery] - https://gerrit.wikimedia.org/r/134819 (owner: QChris)
[13:38:29] <grrrit-wm>	 (CR) QChris: [C: 2 V: 2] "Way better :-)" [analytics/refinery] - https://gerrit.wikimedia.org/r/133425 (owner: Ottomata)
[13:40:43] <grrrit-wm>	 (CR) Ottomata: [V: 2] Limit git-fat to artifacts subdirectory [analytics/refinery] - https://gerrit.wikimedia.org/r/134818 (owner: QChris)
[13:40:51] <grrrit-wm>	 (CR) Ottomata: [V: 2] Whitespace cleanup for gitfat configuration [analytics/refinery] - https://gerrit.wikimedia.org/r/134819 (owner: QChris)
[13:41:20] <ottomata>	 ah! i didn't save my drafts for the review of the other commit
[13:41:35] <grrrit-wm>	 (CR) Ottomata: Adding bin/sequence-file wrapper for refinery-tools (7 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/133519 (owner: Ottomata)
[13:41:55] <qchris>	 Me presses F5 :-D
[13:42:40] <grrrit-wm>	 (PS4) Ottomata: Add bin/sequence-file wrapper for refinery-tools [analytics/refinery] - https://gerrit.wikimedia.org/r/133519
[13:43:23] <grrrit-wm>	 (CR) QChris: [C: -1] Add bin/sequence-file wrapper for refinery-tools (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/133519 (owner: Ottomata)
[13:44:04] <ottomata>	 DUH!
[13:44:14] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/133520/
[13:44:15] <qchris>	 I see ... it's dependent on the refinery/source change.
[13:44:20] <qchris>	 Yes.
[13:45:04] <qchris>	 I looked over that yesterday, and it's fine. Let me just check that the angle brakcets for parameters also wolk nicely on the java libraries for parameter passing.
[13:45:28] <ottomata>	 hmm, ok!
[13:46:11] <ottomata>	 well, actually, I doubt it would make a difference
[13:46:25] <ottomata>	 docopt is the only one where it would matter
[13:46:31] <ottomata>	 as it actually parses the documentation
[13:46:47] <qchris>	 But e.g.: args4j does the opposite.
[13:46:54] <qchris>	 It reads code and prints documentation.
[13:46:58] <ottomata>	 OH
[13:47:01] <ottomata>	 ok let's check it then
[13:47:52] <ottomata>	 results in
[13:47:52] <ottomata>	 "-wrong" is not a valid option
[13:47:52] <ottomata>	  -file FILE : Sets a file if that is present
[13:47:52] <ottomata>	  -name VAL  : Sets a name
[13:47:54] <ottomata>	 http://args4j.kohsuke.org/sample.html
[13:47:58] <ottomata>	 hmm
[13:48:02] <ottomata>	 those are for option docs
[13:48:10] <ottomata>	 but those are arguments to the options
[13:48:11] <ottomata>	 so
[13:48:13] <ottomata>	 captial it is?
[13:48:27] <qchris>	 I guess/hope we can adjust that in args4j.
[13:48:40] <qchris>	 You want this merged before standup?
[13:49:35] <ottomata>	 not necessarily, but why not eh?
[13:49:39] <qchris>	 We'll have to supply metavarwith angle brackets. But that should be ok for now.
[13:49:41] <ottomata>	 i'm changing to captials...
[13:49:44] <ottomata>	 ?
[13:49:45] <qchris>	 Nono.
[13:49:47] <ottomata>	 no?
[13:49:49] <qchris>	 Leave the angle ones.
[13:49:52] <ottomata>	 ok...
[13:49:53] <qchris>	 That's fine.
[13:51:55] <grrrit-wm>	 (CR) QChris: [C: 2 V: 2] "LGTM." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/133520 (owner: Ottomata)
[13:55:09] <ottomata>	 yay :)
[13:55:41] <ottomata>	 I will get you an artifact shortly, and we can merge the other one then
[13:55:57] <qchris>	 Coolio.
[13:59:03] <ottomata>	 qchris, what is 'shim'?
[13:59:11] <ottomata>	 "Maybe something like “percentage_helper”, “percentage_shim”, “shim”, ..."
[13:59:43] <qchris>	 something you force between two things in order to make it work.
[13:59:54] <ottomata>	 haha
[13:59:55] <ottomata>	 hahaha
[14:00:01] <ottomata>	 yes, that is the definition i know too!
[14:00:41] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/134377/
[14:13:40] <grrrit-wm>	 (CR) Milimetric: [WIP] Add cohort class hierarchy, refactor CohortService (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743 (owner: Milimetric)
[14:21:18] <ottomata>	 qchris, milimetric I totally can use the aggregates within the same query
[14:21:22] <ottomata>	 dunno why I thought I couldn't...
[14:21:27] <ottomata>	 so, ja, I think we can just use one query
[14:21:27] <qchris>	 :-D
[14:21:36] <ottomata>	 EVEN with the percentage :p
[14:21:39] <qchris>	 Yay for simplicity
[14:22:04] <qchris>	 Is percentage what we are interested in?
[14:22:13] <qchris>	 (rounding errors, cancellation, ...)
[14:22:22] <ottomata>	 i think both are good
[14:22:24] <ottomata>	 difference and percentage
[14:22:30] <ottomata>	 because the volumne of requests are different for different hosts
[14:22:48] <ottomata>	 so, if i'm going to graph this, its nice to have it uhh...normalized? :p
[14:22:55] <ottomata>	 even though totally denormalizes the table :p
[14:23:17] <qchris>	 evil denormalized tables :-D but ok.
[14:28:55] <grrrit-wm>	 (CR) Ottomata: Add hive/webrequest/presence.hql to help monitor webrequest loss and duplication (12 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/134377 (owner: Ottomata)
[14:29:12] <grrrit-wm>	 (PS2) Ottomata: Add hive/webrequest/presence.hql to help monitor webrequest loss and duplication [analytics/refinery] - https://gerrit.wikimedia.org/r/134377
[14:54:20] <ottomata>	 qchris, hm, maybe we shoudl clean up poms, and import some more stuff into refinery/source before we make a release
[14:54:25] <ottomata>	 meaning, before I add the artifact
[14:54:33] <ottomata>	 for refinery-tools
[14:55:00] <qchris_meeting>	 The artifacts built with the current poms should be fine.
[14:55:14] <qchris_meeting>	 It's just that we can trim some things.
[14:55:29] <qchris_meeting>	 But if you prefer, we can do that beforehand.
[14:56:19] <qchris_meeting>	 About importing more stuff ... sure, we can do. But the current version would already allow to
[14:56:29] <qchris_meeting>	 interact with Snappy SequenceFiles.
[14:56:37] <qchris_meeting>	 That would be helpful already.
[15:01:05] <ottomata>	 yeah, i guess i'm just trying to avoid making more releases than necessary
[15:02:39] <qchris_meeting>	 Agreed. But we'd need a artifact for https://gerrit.wikimedia.org/r/#/c/133519/
[15:02:43] <qchris_meeting>	 wouldn't me?
[15:02:48] <qchris_meeting>	 s/me/we/
[15:03:14] <ottomata>	 yes
[15:03:23] <ottomata>	 i'm saying we'd just wait to merge that too
[15:03:32] <qchris_meeting>	 Ok.
[15:03:39] <qchris_meeting>	 Fine by me :-)
[15:03:50] <qchris_meeting>	 (Just thought you want all those changes merged)
[15:04:25] * qchris_meeting  marks today on the calendar "ottomata hitting the brakes" :-D

[15:04:29] <ottomata>	 haha
[15:04:31] * qchris_meeting  has never seen that before :-D

[15:04:37] <ottomata>	 well, i mean, let's get the source repo all nice
[15:04:38] <ottomata>	 we need to:
[15:05:07] <ottomata>	 - fix up poms
[15:05:08] <ottomata>	 - bring in nuria's ua parser udfs
[15:05:08] <ottomata>	 - geocoding udfs too (?)
[15:05:34] <qchris_meeting>	 Not sure where to draw the line. What about X-Forwarded-For handlng?
[15:07:50] <ottomata>	 Psshhh
[15:08:14] <qchris_meeting>	 Geocoding without X-Forwarded-For handling is pointless :-)
[15:08:26] <qchris_meeting>	 Because you would not know which IP to geocode.
[15:09:36] <ottomata>	 PSsh, this is for prototype only!
[15:10:03] * qchris_meeting  accepts "Psssh" :-)

[15:10:03] <ottomata>	 haha
[16:00:14] <grrrit-wm>	 (PS3) Ottomata: Add hive/webrequest/presence.hql to help monitor webrequest loss and duplication [analytics/refinery] - https://gerrit.wikimedia.org/r/134377
[16:02:57] <grrrit-wm>	 (CR) Nuria: [WIP] Add cohort class hierarchy, refactor CohortService (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743 (owner: Milimetric)
[16:06:17] <ottomata>	 running out to grab lunch, back in a bit
[18:43:56] <grrrit-wm>	 (CR) MarkTraceur: [C: 2] Take sampling factor into account [analytics/multimedia] - https://gerrit.wikimedia.org/r/134065 (owner: Gilles)
[19:15:11] <grrrit-wm>	 (PS3) Milimetric: [WIP] Add cohort class hierarchy, refactor CohortService [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743
[19:58:06] <ottomata>	 heya DarTar, can you join #wikimedia-operations for a sec
[19:58:14] <ottomata>	 i'm asking mutante about the research group thing
[19:58:18] <ottomata>	 Ironholds: yt?
[19:58:58] <Ironholds>	 ottomata, I am!
[19:59:16] <ottomata>	 i'm running that query right now too!
[19:59:19] <ottomata>	 just to find the error
[19:59:29] <ottomata>	 so, i find it very very strange that it takes a while to actually submit that query
[19:59:35] <ottomata>	 i experimented with different versions of it
[19:59:46] <ottomata>	 and noticed the more partitions i worked with, the longer it would take to submit
[20:00:03] <ottomata>	 which i find very strange...as if the hive client has  to process a bunch of metadata before submitting the job?
[20:00:05] <ottomata>	 dunno...
[20:00:11] <Ironholds>	 yeah, I have no idea.
[20:00:14] <Ironholds>	 The query is actually now running
[20:00:19] <Ironholds>	 I'll let you know if it explodes
[20:00:35] <ottomata>	 wait mine is too!
[20:00:35] <Ironholds>	 but it's 53/4 percent of the way through the subquery without balking.
[20:00:38] <Ironholds>	 That makes me optimistic.
[20:00:46] <ottomata>	 oh of the mapper?
[20:00:51] <Ironholds>	 the only question is whether it'll hit 100/100 and/or die on the second MR task
[20:00:56] <Ironholds>	 53 map, 4 reduce
[20:01:09] <ottomata>	 oh, hm, mine has been running for like an hour
[20:01:11] <ottomata>	 and it is only at 2%
[20:01:17] <ottomata>	  i will kill mine then
[20:01:20] <Ironholds>	 probably because it's running next to mine
[20:01:21] <Ironholds>	 good luck!
[20:01:21] <ottomata>	 if you are going to get the error output from yours
[20:01:26] <ottomata>	 gaha
[20:01:28] <ottomata>	 ha
[20:01:42] <Ironholds>	 yeah, it's pretty much impossible, I've found, to elegantly kill queries from the terminal or run multiple MR tasks at once.
[20:01:59] <Ironholds>	 I tried, a few weeks ago, running 3 tasks simultaneously. Bad idea.
[20:01:59] <Ironholds>	 They were actually running a deficit on the reduce portion.
[20:02:04] <Ironholds>	 That is: it took more than a second to add a second of CPU time.
[20:02:16] <ottomata>	 ha, hm
[20:02:24] <ottomata>	 you can kill queries!
[20:02:32] <ottomata>	 yarn application -kill application_1387838787660_12241
[20:02:37] <Ironholds>	 ahhh
[20:02:47] <Ironholds>	 ctrl+cing is not somrthing hive likes
[20:02:50] <ottomata>	 just killed mine
[20:02:51] <ottomata>	 yeah, it doesn't
[20:02:57] <ottomata>	 because at that point hive isn't controlling the job anymore
[20:03:01] <ottomata>	 its a distributed hadoop yarn application
[20:03:21] <ottomata>	 the only reason you get output at all is because your hive process is still connected to the job somehow
[20:03:24] <ottomata>	 maybe it just polls, dunno
[20:03:36] <Ironholds>	 ahhhhh
[20:03:43] <Ironholds>	 okay! We should absolutely note that somewhere.
[20:03:46] <ottomata>	 so if you ctrl-c, you just kill the interface
[20:03:52] <Ironholds>	 I need the breathing space to write fuller hive documentation
[20:04:05] <ottomata>	 i'll add that to the hive page
[20:08:05] <ottomata>	 Ironholds: https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Hive#Killing_a_running_query
[20:08:11] <Ironholds>	 danke!
[20:37:49] <grrrit-wm>	 (PS4) Milimetric: [WIP] Add cohort class hierarchy, refactor CohortService [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743
[21:41:08] <grrrit-wm>	 (PS5) Milimetric: [WIP] Add cohort class hierarchy, refactor CohortService [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743
[21:42:14] <grrrit-wm>	 (PS1) Milimetric: Fix bad file_manager creation [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134951
[21:44:05] <grrrit-wm>	 (PS6) Milimetric: [WIP] Add cohort class hierarchy, refactor CohortService [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743
[22:20:16] <grrrit-wm>	 (PS7) Milimetric: [WIP] Add cohort class hierarchy, refactor CohortService [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134743