[00:39:52] <wikibugs>	 Analytics / Wikimetrics: Cannot validate a new cohort - https://bugzilla.wikimedia.org/71099#c1 (nuria) Very sorry about this. The labs cluster had some maintenance done this weekend and seems like things are no working as they should quite yet. Once issues with labs are solved we will update this bug,...
[01:33:31] <grrrit-wm>	 (CR) Nuria: "I reviewed the code but could not test in labs due to the db situation." (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/161521 (owner: Milimetric)
[03:45:04] <grrrit-wm>	 (PS3) QChris: Add Oozie setup to aggregate Webstatscollector pagecount files [analytics/refinery] - https://gerrit.wikimedia.org/r/160636
[03:45:06] <grrrit-wm>	 (PS1) QChris: Stop tying Oozie's “data_directory” to webrequests table [analytics/refinery] - https://gerrit.wikimedia.org/r/161904
[03:45:08] <grrrit-wm>	 (PS1) QChris: Move Oozie's “marking directory done” into separate workflow [analytics/refinery] - https://gerrit.wikimedia.org/r/161905
[03:45:10] <grrrit-wm>	 (PS1) QChris: Add Oozie setup to rendering webstatscollector files [analytics/refinery] - https://gerrit.wikimedia.org/r/161906
[07:13:38] <wikibugs>	 Analytics / Tech community metrics: Graphs for median/average should report absolute numbers - https://bugzilla.wikimedia.org/66266#c4 (Quim Gil) NEW>ASSI a:Alvaro Looks good, thank you!
[07:20:07] <wikibugs>	 Analytics / Tech community metrics: Code review metrics should not include [WIP] changesets - https://bugzilla.wikimedia.org/66283#c4 (Quim Gil) In case it's useful, now apps/android/commons made it to the top of the list, but it has only two WIP open changesets:  https://gerrit.wikimedia.org/r/#/q/sta...
[11:33:10] <wikibugs>	 Analytics / General/Unknown: Packetloss_Average alarm on udp2log machines on 2014-09-20 - https://bugzilla.wikimedia.org/71116 (christian) NEW p:Unprio s:normal a:None There have been Packetloss_Average alerts on 2014-09-20 for a few minutes on erbium and oxygen [1].  What happened and how...
[11:36:53] <wikibugs>	 Analytics / General/Unknown: Packetloss_Average alarm on udp2log machines on 2014-09-20 - https://bugzilla.wikimedia.org/71116#c1 (christian) Ops reported an ULSO outage [1] that matches the time period, and Ops said to have a proper incident report today (2014-09-22).  Once that is out, we'll see how...
[12:04:40] <grrrit-wm>	 (CR) QChris: "> > However, we do it nonetheless, as webstatscollector is doing it too." [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)
[14:01:04] <qchris>	 ottomata: Standup time :-)
[14:01:08] <ottomata>	 ooF
[14:01:13] <qchris>	 sneaky meeting!
[14:23:49] <grrrit-wm>	 (PS3) Milimetric: Update NamespaceEdits metric [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/161470 (https://bugzilla.wikimedia.org/71008)
[14:24:13] <grrrit-wm>	 (PS3) Milimetric: Update PagesCreated metric [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/161472
[14:24:42] <grrrit-wm>	 (PS4) Milimetric: Update PagesCreated metric [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/161472 (https://bugzilla.wikimedia.org/71009)
[14:25:24] <grrrit-wm>	 (PS2) Milimetric: Add Rolling Recurring Old Active Editor [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/161521 (https://bugzilla.wikimedia.org/69569)
[14:52:38] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Stop tying Oozie's “data_directory” to webrequests table [analytics/refinery] - https://gerrit.wikimedia.org/r/161904 (owner: QChris)
[14:54:26] <grrrit-wm>	 (CR) Ottomata: Move Oozie's “marking directory done” into separate workflow (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/161905 (owner: QChris)
[14:57:06] <grrrit-wm>	 (PS2) QChris: Add Oozie setup to rendering webstatscollector files [analytics/refinery] - https://gerrit.wikimedia.org/r/161906
[14:57:08] <grrrit-wm>	 (PS2) QChris: Move Oozie's “marking directory done” into separate workflow [analytics/refinery] - https://gerrit.wikimedia.org/r/161905
[14:57:10] <grrrit-wm>	 (PS4) QChris: Add Oozie setup to aggregate Webstatscollector pagecount files [analytics/refinery] - https://gerrit.wikimedia.org/r/160636
[14:57:50] <grrrit-wm>	 (CR) QChris: Move Oozie's “marking directory done” into separate workflow (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/161905 (owner: QChris)
[15:06:16] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Move Oozie's “marking directory done” into separate workflow [analytics/refinery] - https://gerrit.wikimedia.org/r/161905 (owner: QChris)
[15:06:29] <grrrit-wm>	 (CR) Ottomata: Add Oozie setup to aggregate Webstatscollector pagecount files (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)
[15:08:50] <ottomata>	 whoaa oozie switch
[15:08:53] <ottomata>	 not seen this one yet...
[15:09:16] <qchris>	 :-D
[15:12:59] <ottomata>	 whoa, there is a fork!
[15:13:01] <ottomata>	 that's cool
[15:13:10] <ottomata>	 cool
[15:23:10] <ottomata>	 qchris: quick q, your hive query right now runs as two queries
[15:23:14] <ottomata>	 one for the .mw counts
[15:23:20] <qchris>	 right.
[15:23:22] <ottomata>	 won't that create multiple output files?
[15:23:28] <qchris>	 Yes.
[15:23:29] <ottomata>	 (i'm reading your prepare_job_output change now)
[15:23:55] <qchris>	 But there is an order by when preparing the {prajec,page}counts.
[15:24:02] <qchris>	 s/prajec/project/
[15:24:07] <qchris>	 This order by merges them again.
[15:24:33] <ottomata>	 ah select from destination table
[15:24:34] <ottomata>	 hm ok
[15:25:44] <ottomata>	 confused about how that works, but i will come back to it
[15:26:12] <qchris>	 k
[15:27:46] <ottomata>	 another quick q:
[15:28:02] <ottomata>	 woudl it not be simpler to write a shell wrapper for this, rather than all the oozie cases?
[15:28:06] <ottomata>	 just to get different error messages?
[15:28:24] <ottomata>	 i understand not shelling out for each of the fs checks
[15:28:28] <ottomata>	 but what abou tthe whole thing?
[15:28:32] <qchris>	 I thought a lot about that.
[15:28:43] <qchris>	 Did you read the comment in the middle of the switch?
[15:28:45] <ottomata>	 yes
[15:28:52] <ottomata>	 so
[15:29:06] <qchris>	 So to me, output parsing of "hdfs dfs -ls" likely to break sooner.
[15:29:06] <ottomata>	 i think your cases are good, in that shelling out in each one to hdfs -ls would be cumbersome
[15:29:19] <ottomata>	 but i mean a wrapper for the whole thing
[15:29:38] <qchris>	 When you read the help page of "hdfs dfs -ls", the given description already does not match the actual output.
[15:29:38] <ottomata>	 that takes in source_directory, target_file, expecting_ending, etc
[15:29:41] <ottomata>	 as parameters
[15:30:04] <grrrit-wm>	 (CR) QChris: Add Oozie setup to aggregate Webstatscollector pagecount files (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)
[15:30:31] <qchris>	 By wrapper, you mean a ...say ... bash script. Right?
[15:30:43] <ottomata>	 yes
[15:30:46] <ottomata>	 or python, or whatever
[15:30:58] <ottomata>	 haha, python + snakebite :p
[15:30:59] <ottomata>	 jk
[15:31:04] <ottomata>	 that would be complicated too :/
[15:31:09] <qchris>	 How would you parse the output of "hdfs dfs -ls" into something like "find"?
[15:31:13] <ottomata>	 but would sidestep the issue of hdfs -ls
[15:31:18] <ottomata>	 https://github.com/spotify/snakebite
[15:31:22] <qchris>	 By just splitting?
[15:31:46] <ottomata>	 hm?
[15:31:56] <ottomata>	 naw, you'd just be using an API, rather than parsing text output
[15:31:57] <ottomata>	 http://spotify.github.io/snakebite/
[15:32:22] <qchris>	 Sorry. The latest question was not meant about snakebite, but about "hdfs dfs -ls".
[15:32:23] <ottomata>	 At Spotify we use the http://github.com/spotify/luigi that relies on doing a lot of existence checks and moving data around in HDFS. And since callinghadoop from python is expensive, we decided to write a pure python HDFS client that only relies on protobuf.
[15:32:57] <ottomata>	 ah, my (halfhearted) idea is to  write a python wrapper that uses snakebite to do this
[15:33:29] <qchris>	 That is heavy tooling for the tiny use case we currently have.
[15:33:52] <qchris>	 Could we kick that down the road, until we have more users of it?
[15:34:05] <ottomata>	 yeah for sure
[15:34:09] <ottomata>	 i agree
[15:34:18] <ottomata>	 we'd have to package that thing up to, that would delay us a lot
[15:34:30] <qchris>	 Everything is kept in a separate workflow, so improving that workflow, would improve all the uses of it.
[15:34:33] <ottomata>	 yeah
[15:34:45] <qchris>	 Cool.
[15:36:29] <ottomata>	 ok, another q
[15:36:46] <ottomata>	 can't the webstatscollector table data itself be used for the pagecount file?
[15:36:54] <ottomata>	 if we store it as textfile tsv?
[15:37:50] <qchris>	 A hour in the webstatscollector table might be represented by more than 1 file (*), so we need to merge them into a single file.
[15:37:59] <ottomata>	 AHHHH
[15:38:03] <qchris>	 (*) currently 2 files. But once you mess with the table, the number increases.
[15:38:22] <ottomata>	 ahhh ok, my previous question about hte two queries makes sense now
[15:38:22] <ottomata>	 ok ok
[15:38:24] <ottomata>	 got it.
[15:38:43] <qchris>	 Awesome.
[15:43:01] <grrrit-wm>	 (CR) Ottomata: Add Oozie setup to rendering webstatscollector files (3 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/161906 (owner: QChris)
[15:51:10] <grrrit-wm>	 (CR) Ottomata: Add Oozie setup to aggregate Webstatscollector pagecount files (5 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)
[15:57:46] <ottomata>	 qchris: am going into a meeting now, and then again in 2 hours
[15:57:57] <qchris>	 I am there already :-)
[15:57:58] <ottomata>	 will be kinda avail to keep reviewing in 1 hour for 1 hour :)
[15:58:03] <ottomata>	 oh yeah you are in that one too! :p
[16:00:23] <ottomata>	 oh qchris, just thought of something re kafkatee,
[16:00:29] <ottomata>	 fundraising needs it
[16:00:33] <ottomata>	 or probably
[16:00:36] <ottomata>	 ah, meeting starting ok later
[17:26:54] <grrrit-wm>	 (CR) QChris: Add Oozie setup to aggregate Webstatscollector pagecount files (5 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)
[17:40:55] <wikibugs>	 Analytics / Wikistats: One column header in browsers per country report only contains '---' - https://bugzilla.wikimedia.org/71127 (Erik Zachte) NEW p:Unprio s:minor a:None http://stats.wikimedia.org/wikimedia/squids/SquidReportCountryBrowser.htm  reported by Atul Vaidya
[17:43:28] <grrrit-wm>	 (CR) QChris: Add Oozie setup to rendering webstatscollector files (3 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/161906 (owner: QChris)
[19:12:52] <wikibugs>	 Analytics / Wikimetrics: Cannot validate a new cohort - https://bugzilla.wikimedia.org/71099#c2 (nuria) Issues in production are solved, we are taking a few minutes to make sure everything is OK and wikimetrics should be back online in a few minutes.
[19:21:07] <wikibugs>	 Analytics / Tech community metrics: Code review metrics should not include [WIP] changesets - https://bugzilla.wikimedia.org/66283#c5 (Alvaro) REOP>PATC Quim, WIP reviews are now filtered.  And as expected, the two repos above don't appear anymore.
[20:52:19] <ottomata>	 qchris: quick comment about the _sum vs _count
[20:52:25] <ottomata>	 _count is a suffix, as is _sum, right?
[20:52:31] <ottomata>	 so
[20:52:45] <ottomata>	 view_count and response_size_sum kinda make sense...although I don't love response_size_sum
[20:52:54] <ottomata>	 oo size_total
[20:52:57] <ottomata>	 would be fine with me
[20:53:06] <ottomata>	 response_size_total
[20:53:10] <qchris>	 Mhmm.
[20:53:17] <qchris>	 So the two would get a different suffix?
[20:53:27] <ottomata>	 yes, aren't the computed differently...?
[20:53:33] <ottomata>	 checking...
[20:53:47] <qchris>	 No. SUM(...)
[20:53:50] <ottomata>	 yeah
[20:53:51] <ottomata>	      COUNT(*) count,
[20:53:51] <ottomata>	 javascript:;
[20:53:51] <ottomata>	         SUM(response_size) response_size
[20:54:18] <ottomata>	 ah
[20:54:19] <qchris>	 Oh. OH!
[20:54:23] <ottomata>	 qchis, in sequence stats table
[20:54:23] <qchris>	 You're right.
[20:54:26] <ottomata>	 we prefixed
[20:54:27] <ottomata>	 count_*
[20:54:30] <ottomata>	 so
[20:54:46] <ottomata>	 count_views
[20:54:46] <ottomata>	 (sum|total)_response_size
[20:54:47] <ottomata>	 or maybe
[20:54:53] <ottomata>	 bytes_served
[20:54:53] <ottomata>	 ?
[20:55:11] <qchris>	 the column in hive is called "response_size"
[20:55:14] <ottomata>	 although, i do like sum|total_response_size, as...y
[20:55:15] <ottomata>	 eah
[20:55:18] <ottomata>	 what i was gonna say. :)
[20:57:03] <qchris>	 even if they are computed different ... we might also express the count via sum.
[20:57:11] <qchris>	 For me it's confusing to call them differently.
[20:57:19] <qchris>	 But meh.
[20:57:50] <qchris>	 So ... "count_views" and "sum_response_size?
[20:58:05] <qchris>	 "count_views" and "totol_response_size?
[21:03:24] <ottomata>	 i kinda like total_response_size
[21:03:27] <ottomata>	 very descriptive
[21:04:34] <qchris>	 Ok.
[21:04:45] <qchris>	 Then "count_views" and "total_response_size" it is.
[21:05:34] <ottomata>	 cool.
[21:05:45] <qchris>	 About the other column names ...
[21:06:01] <qchris>	 Should we discuss right away too, or do you prefer to do it in gerrit?
[21:07:07] <ottomata>	 lemme keep going in gerrit, am responding to recent comments now.
[21:07:29] <qchris>	 ok.
[21:08:06] <grrrit-wm>	 (CR) QChris: Add Oozie setup to aggregate Webstatscollector pagecount files (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)
[21:12:10] <wikibugs>	 Analytics / Wikimetrics: Story: AnalyticsEng uses connection pooling on database URL - https://bugzilla.wikimedia.org/71140 (Dan Andreescu) NEW p:Unprio s:normal a:None Currently we're hitting a different connection string for each different labsdb database name.  We could change this to a...
[21:15:12] <grrrit-wm>	 (CR) Ottomata: Add Oozie setup to aggregate Webstatscollector pagecount files (5 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)
[21:19:17] <ottomata>	 wait, qchris, explain to me why we need this special prepare hdfs mv step?
[21:19:35] <ottomata>	 because INSERT OVERWRITE DIRECTORY might generate multiple files?
[21:19:43] <qchris>	 Let me reread the code.
[21:21:01] <ottomata>	 i.e., in render_hourly_files/workflow.xml
[21:21:04] <ottomata>	 why can't we just do
[21:21:12] <qchris>	 https://gerrit.wikimedia.org/r/#/c/161906/2/oozie/util/prepare_job_output_for_plainfs_rsync/workflow.xml
[21:21:16] <ottomata>	 <param>destination_directory=path/to/final/output/directory
[21:21:17] <ottomata>	 ?
[21:21:36] <ottomata>	 for the     <action name="extract_data_into_single_file">
[21:21:36] <ottomata>	  action
[21:22:46] <qchris>	 I guess I do not understand the question.
[21:22:52] <qchris>	 Sorry.
[21:23:26] <qchris>	 So are you asking why extract_data_into_single_file is a hive query instead of just a set of hdfs commands?
[21:23:38] <ottomata>	 uh no
[21:23:46] <ottomata>	 i am asking why the hive query doesn't just insert into the final directory
[21:23:54] <qchris>	 Ah!
[21:24:03] <ottomata>	 why do we need the     <action name="move_data_to_rsync_directory">
[21:24:03] <ottomata>	  
[21:24:03] <ottomata>	 action
[21:24:04] <ottomata>	 ?
[21:24:11] <qchris>	 Because a job output is always a directory.
[21:24:20] <ottomata>	 hive job?
[21:24:21] <ottomata>	 hm
[21:24:22] <ottomata>	 hm
[21:24:23] <ottomata>	 hm
[21:24:24] <ottomata>	 right
[21:24:25] <ottomata>	 hm
[21:24:27] <qchris>	 And the file within that directory is typically 00000_0.gz
[21:24:34] <qchris>	 It's hard to control that reliably.
[21:24:39] <ottomata>	 GOT it.
[21:24:57] <ottomata>	 ok, the comment I have then
[21:25:07] <ottomata>	 i don't like calling this the 'rsync directory'
[21:25:08] <ottomata>	 or wahtever
[21:25:13] <ottomata>	 no reason to tie it to a tool
[21:25:16] <qchris>	 :-D
[21:25:33] <ottomata>	 arhive?
[21:25:34] <ottomata>	 archive?
[21:25:38] <ottomata>	 archive_directory?
[21:25:47] <ottomata>	 prepare_job_output_for_directory
[21:25:48] <ottomata>	 ?
[21:25:49] <ottomata>	 oops
[21:25:53] <ottomata>	 prepare_job_output_for_archive_directory
[21:25:53] <ottomata>	 ?
[21:25:54] <qchris>	 archive_directory sounds good to me.
[21:26:28] <qchris>	 What about "prepare_job_output_for_archive_directory" -> "archive_job_output" ?
[21:26:35] <ottomata>	 +1
[21:26:56] <qchris>	 Ok. I will rework the patches then.
[21:27:01] <ottomata>	 k
[21:27:03] <qchris>	 Thanks for your reviews!
[21:27:12] <grrrit-wm>	 (CR) Ottomata: Add Oozie setup to rendering webstatscollector files (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/161906 (owner: QChris)
[21:27:31] <ottomata>	 thank you for your work!
[21:27:35] <ottomata>	 looking as awesome as ever :)
[21:29:49] <ottomata>	 ok, out for the eve, thikn I"m going to go get a new iphone, ok byyeyeyeye
[21:29:57] <qchris>	 :-)
[21:30:01] <qchris>	 have fun.
[21:31:54] <grrrit-wm>	 (CR) QChris: Add Oozie setup to rendering webstatscollector files (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/161906 (owner: QChris)
[22:07:51] <wikibugs>	 Analytics / Tech community metrics: Code review metrics should not include [WIP] changesets - https://bugzilla.wikimedia.org/66283#c6 (Quim Gil) PATC>RESO/FIX I've checked the top 10 and everything was perfect. Thank you!
[22:33:44] <grrrit-wm>	 (CR) QChris: Add Oozie setup to aggregate Webstatscollector pagecount files (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/160636 (owner: QChris)