[05:55:18] <grrrit-wm>	 (CR) Gilles: [C: 2] Code hygiene: pass flake8 python linter [analytics/multimedia] - https://gerrit.wikimedia.org/r/134128 (owner: Hashar)
[05:55:50] <grrrit-wm>	 (CR) Gilles: [V: 2] Code hygiene: pass flake8 python linter [analytics/multimedia] - https://gerrit.wikimedia.org/r/134128 (owner: Hashar)
[05:56:23] <grrrit-wm>	 (CR) Gilles: [C: 2 V: 2] Tox environement to run flake8 python linter [analytics/multimedia] - https://gerrit.wikimedia.org/r/134144 (owner: Hashar)
[08:01:07] <grrrit-wm>	 (CR) Hashar: "recheck" [analytics/multimedia] - https://gerrit.wikimedia.org/r/134065 (owner: Gilles)
[09:02:42] <grrrit-wm>	 (PS1) Hashar: Tox configuration to run flake8 (python linter) [analytics/wp-zero] - https://gerrit.wikimedia.org/r/134314
[09:05:06] <grrrit-wm>	 (CR) Hashar: "recheck" [analytics/wp-zero] - https://gerrit.wikimedia.org/r/134314 (owner: Hashar)
[09:07:31] <grrrit-wm>	 (CR) Hashar: "The job fail because the repository has a bunch of linting errors :-]" [analytics/wp-zero] - https://gerrit.wikimedia.org/r/134314 (owner: Hashar)
[09:30:36] <grrrit-wm>	 (PS1) Hashar: Lint: allow commenting of code [analytics/wp-zero] - https://gerrit.wikimedia.org/r/134320
[09:30:38] <grrrit-wm>	 (PS1) Hashar: Lint: pass pep8 checks [analytics/wp-zero] - https://gerrit.wikimedia.org/r/134321
[09:30:40] <grrrit-wm>	 (PS1) Hashar: Lint: remove unused imports [analytics/wp-zero] - https://gerrit.wikimedia.org/r/134322
[09:31:28] <grrrit-wm>	 (CR) Hashar: "The flake8 Jenkins job is now passing \O/" [analytics/wp-zero] - https://gerrit.wikimedia.org/r/134322 (owner: Hashar)
[11:42:13] <grrrit-wm>	 (CR) Ottomata: Adding refinery-tools and pom.xml content (6 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/133520 (owner: Ottomata)
[11:43:38] <grrrit-wm>	 (CR) Ottomata: "I did not try to run any mvn commands for this commit. It works fine in the next one though, as the parent pom is no longer empty, and it" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/133257 (owner: Ottomata)
[11:43:53] <grrrit-wm>	 (CR) Ottomata: "OOF, thanks for catching that..." [analytics/refinery] - https://gerrit.wikimedia.org/r/133425 (owner: Ottomata)
[11:44:13] <grrrit-wm>	 (PS2) Ottomata: Adding refinery-tools and pom.xml content [analytics/refinery/source] - https://gerrit.wikimedia.org/r/133520
[11:45:03] <grrrit-wm>	 (CR) Ottomata: "For safety's sake, I'm going to abandon this and create a new commit." [analytics/refinery] - https://gerrit.wikimedia.org/r/133425 (owner: Ottomata)
[11:46:30] <grrrit-wm>	 (CR) Ottomata: Initial repository layout (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/133425 (owner: Ottomata)
[11:46:47] <grrrit-wm>	 (Abandoned) Ottomata: Initial repository layout [analytics/refinery] - https://gerrit.wikimedia.org/r/133425 (owner: Ottomata)
[11:49:14] <grrrit-wm>	 (Restored) Ottomata: Initial repository layout [analytics/refinery] - https://gerrit.wikimedia.org/r/133425 (owner: Ottomata)
[11:54:05] <grrrit-wm>	 (PS3) Ottomata: Initial repository layout [analytics/refinery] - https://gerrit.wikimedia.org/r/133425
[11:56:43] <grrrit-wm>	 (CR) Ottomata: "Ok, qchris, how's that?" [analytics/refinery] - https://gerrit.wikimedia.org/r/133425 (owner: Ottomata)
[11:59:19] <grrrit-wm>	 (PS2) Ottomata: Adding bin/sequence-file wrapper for refinery-tools [analytics/refinery] - https://gerrit.wikimedia.org/r/133519
[12:02:32] <grrrit-wm>	 (PS1) Hashar: Tox configuration to run flake8 (python linter) [analytics/geowiki] - https://gerrit.wikimedia.org/r/134334
[12:09:21] <grrrit-wm>	 (PS3) Ottomata: Adding bin/sequence-file wrapper for refinery-tools [analytics/refinery] - https://gerrit.wikimedia.org/r/133519
[12:10:11] <grrrit-wm>	 (CR) Hashar: "recheck" [analytics/geowiki] - https://gerrit.wikimedia.org/r/134334 (owner: Hashar)
[12:13:35] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] "Merging this. (some) poms are added in the next commit." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/133257 (owner: Ottomata)
[12:15:46] <grrrit-wm>	 (PS1) Hashar: Lint setup.py [analytics/geowiki] - https://gerrit.wikimedia.org/r/134339
[12:15:48] <grrrit-wm>	 (PS1) Hashar: Replace git URL to git.wikimedia.org [analytics/geowiki] - https://gerrit.wikimedia.org/r/134340
[12:19:01] <grrrit-wm>	 (PS2) Hashar: Tox configuration to run flake8 (python linter) [analytics/geowiki] - https://gerrit.wikimedia.org/r/134334
[12:19:25] <grrrit-wm>	 (CR) Hashar: "Ignore lines being too long and comment starting with '# '" [analytics/geowiki] - https://gerrit.wikimedia.org/r/134334 (owner: Hashar)
[12:26:42] <grrrit-wm>	 (PS1) Hashar: Lint geowiki/mysql_config.py [analytics/geowiki] - https://gerrit.wikimedia.org/r/134342
[12:32:49] <grrrit-wm>	 (PS1) Hashar: Lint geowiki/geo_coding.py [analytics/geowiki] - https://gerrit.wikimedia.org/r/134345
[12:39:52] <grrrit-wm>	 (PS1) Hashar: Lint geowiki/process_data.py [analytics/geowiki] - https://gerrit.wikimedia.org/r/134347
[12:41:49] <grrrit-wm>	 (PS2) Hashar: Lint setup.py [analytics/geowiki] - https://gerrit.wikimedia.org/r/134339
[12:41:57] <grrrit-wm>	 (PS2) Hashar: Replace git URL to git.wikimedia.org [analytics/geowiki] - https://gerrit.wikimedia.org/r/134340
[12:43:36] <hashar>	 I have added flake8 jobs to some of your repositories (geowikip, wp-zero and multimedia) :D
[13:34:08] <milimetric>	 thanks hashar, saw that
[13:34:34] <hashar>	 I have submitted a bunch of follow up patch to pass flake8
[13:35:21] <hashar>	 s/submitted/proposed/
[14:13:46] <grrrit-wm>	 (PS1) Milimetric: Clean up imports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134356
[14:13:48] <grrrit-wm>	 (PS1) Milimetric: Fix type-o [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134357
[15:05:21] <ottomata>	 Ironholds: helloooo
[15:05:27] <ottomata>	 what'sup?
[16:12:39] <grrrit-wm>	 (PS1) Ottomata: Add hive/webrequest/presence.hql to help monitor webrequest loss and duplication [analytics/refinery] - https://gerrit.wikimedia.org/r/134377
[16:50:39] <Ironholds>	 ottomata, yo :)
[16:50:44] <Ironholds>	 sorry, was/sort of am in standup.
[16:50:48] <Ironholds>	 What's occurring?
[16:51:43] <ottomata>	 oh was just pinging about your hive probs
[16:51:48] <ottomata>	 i'm running your other query now
[16:51:55] <ottomata>	 you are trying to return 5million results though...
[16:55:51] <ottomata>	 wait, Ironholds, why the subquery there?
[16:56:31] <Ironholds>	 ottomata, because I want 5m randomly selected rows
[16:56:46] <Ironholds>	 and, yes. Should I not be? ;).
[16:57:02] <ottomata>	 https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Sampling
[16:57:50] <Ironholds>	 neat!
[16:58:10] <Ironholds>	 and WHERE clauses still apply?
[16:58:17] <ottomata>	 i've never done it, but i guess so!
[16:58:20] <Ironholds>	 the only bugbear is it doesn't allow a raw count
[16:59:44] <Ironholds>	 wait, no, it does! cool!
[17:01:05] <Ironholds>	 okay, let's see if that helps.
[17:01:16] <Ironholds>	 it shouldn't, because a query very much like this has worked before, but.
[17:01:31] <ottomata>	 ok, yeah i'm not saying there isn't a problem
[17:01:39] <ottomata>	 you've gotten hive to return you 5M rows on the CLI before?
[17:04:23] <Ironholds>	 yerp
[17:04:53] <Ironholds>	 the mobile sessions work
[17:05:10] <Ironholds>	 augh. ottomata, we're on Cloudera 4, aren't we.
[17:05:33] <ottomata>	 yup
[17:05:36] <ottomata>	 why?
[17:06:04] <Ironholds>	 support for TABLESAMPLE(N ROWS) is CDH5 up.
[17:06:33] <Ironholds>	 okay, retrying with a smaller number to see the class of problem we're dealing with. Let's see if that helps.
[17:07:35] <ottomata>	 ah but Ironholds we have hive 0.10.0
[17:07:45] <ottomata>	 ah
[17:07:47] <ottomata>	 the JIRA says 		Fix Version/s:
[17:07:47] <ottomata>	 https://issues.apache.org/jira/browse/HIVE/fixforversion/12323587
[17:07:53] <ottomata>	 but the apache wiki says 0.10.0
[17:07:54] <ottomata>	 ah well
[17:08:24] <Ironholds>	 okay, it's giving me the same heap space error on 1m rows.
[17:08:31] <Ironholds>	 And that number I /know/ I've used before ;p
[17:08:44] <Ironholds>	 in fact, more than 1m, because it was all observations associated with 500k IPs.
[17:13:29] <ottomata>	 HMMmmmMMm
[17:20:42] <Ironholds>	 huh!
[17:20:46] <Ironholds>	 you know what I think it might be?
[17:20:57] <Ironholds>	 the rlikes are causing it to go "bah, I don't have enough capacity for this! die! die!"
[17:21:12] <Ironholds>	 the first entries in the traceback are:
[17:21:13] <Ironholds>	 at java.util.Arrays.copyOf(Arrays.java:2367)
[17:21:13] <Ironholds>	 	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
[17:21:13] <Ironholds>	 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
[17:21:13] <Ironholds>	 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
[17:21:14] <Ironholds>	 	at java.lang.StringBuffer.append(StringBuffer.java:237)
[17:21:15] <Ironholds>	 	at
[17:21:17] <Ironholds>	 etc etc
[17:22:09] <ottomata>	 Oliver, I got it a bit farther
[17:22:09] <ottomata>	 by setting
[17:22:09] <grrrit-wm>	 (CR) Milimetric: [V: 1] Add hive/webrequest/presence.hql to help monitor webrequest loss and duplication (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/134377 (owner: Ottomata)
[17:22:09] <ottomata>	 export HADOOP_HEAPSIZE=1024
[17:22:21] <ottomata>	 but, not sure, there are some other weird things too
[17:22:25] <Ironholds>	 hrm
[17:22:50] <Ironholds>	 if we can't MapReduce regexes we're kind of boned for...a lot of stuff.
[17:27:49] <nuria>	 ironholds: we did a bunch of queries with regexes for the glam folks before
[17:28:07] <Ironholds>	 hmhmnmnmn.
[17:28:19] <Ironholds>	 so, then, nothing in this query has not been done before
[17:28:22] <Ironholds>	 it's just exploding.
[17:28:27] <Ironholds>	 because...god hates me?
[17:32:02] <ottomata>	 i think god hates you, and I think there is a little naughty directory somewhere it shouldn't be in the data
[17:32:09] <ottomata>	 Job Submission failed with exception 'java.io.FileNotFoundException(Path is not a file: /wmf/data/external/webrequest/webrequest_bits/hourly/2014/05/15/08/08
[17:32:19] <ottomata>	 we will see...trying again
[17:40:19] <nuria>	 ironholds: you can checkout glam seklects here: https://commons.m.wikimedia.org/wiki/Commons:GLAMwiki_Toolset_Project/NARA_analytics_pilot
[17:40:24] <nuria>	 *selects
[17:43:46] <grrrit-wm>	 (PS1) Terrrydactyl: Capitalized User for dummy users. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134395
[17:44:04] <ottomata>	 Ironholds:
[17:44:06] <ottomata>	 try it now
[17:44:13] <ottomata>	 first, without setting HADOOP_HEAPSIZE
[17:44:15] <ottomata>	 if that fails you
[17:44:16] <ottomata>	 then set it
[17:44:28] <ottomata>	 export HADOOP_HEAPSIZE=1024; hive ...
[17:44:47] <ottomata>	 you could do larger than that if you need
[17:44:48] <ottomata>	 i think it might run now...
[17:45:08] <ottomata>	 i'm going to run for a while, maybe someone will be in paris to play polo with me...doubtful since it rained today
[17:45:09] <ottomata>	 laaters!
[17:48:37] <grrrit-wm>	 (PS1) Nuria: Can upload names with utf-8 characters via "Paste Usernames" textbox [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134397 (https://bugzilla.wikimedia.org/64893)
[17:49:11] <milimetric>	 good luck ottomata
[17:50:04] <Ironholds>	 ottomata, totaly
[17:53:37] <grrrit-wm>	 (CR) Nuria: [C: 2] Clean up imports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134356 (owner: Milimetric)
[18:01:21] <grrrit-wm>	 (CR) Milimetric: Can upload names with utf-8 characters via "Paste Usernames" textbox (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134397 (https://bugzilla.wikimedia.org/64893) (owner: Nuria)
[18:03:27] <grrrit-wm>	 (CR) Nuria: [C: 2] Fix type-o [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134357 (owner: Milimetric)
[18:03:40] <grrrit-wm>	 (CR) Milimetric: [V: 2] Capitalized User for dummy users. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134395 (owner: Terrrydactyl)
[18:03:50] <grrrit-wm>	 (PS2) Milimetric: Capitalized User for dummy users. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134395 (owner: Terrrydactyl)
[18:03:58] <grrrit-wm>	 (CR) Milimetric: [C: 2] Capitalized User for dummy users. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134395 (owner: Terrrydactyl)
[18:05:51] <Ironholds>	 ottomata, failed without the heapsize change
[18:05:52] <Ironholds>	 trying again
[18:15:34] <Ironholds>	 ottomata, well, it's at least /running/
[18:15:40] <Ironholds>	 the slowest damnt hing I've ever seen, but it runs ;p
[20:05:48] <grrrit-wm>	 (PS1) Milimetric: Refactor cohort methods into service [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/134480
[21:16:48] <superm401>	 ottomata, can you unabandom https://gerrit.wikimedia.org/r/#/c/49678/ ?  It looks like it is no longer blocked, and a few people are interested.
[21:44:42] <grrrit-wm>	 (CR) Gergő Tisza: [C: 1] Take sampling factor into account [analytics/multimedia] - https://gerrit.wikimedia.org/r/134065 (owner: Gilles)
[22:42:21] <grrrit-wm>	 (PS3) Terrrydactyl: [WIP] Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091
[22:42:39] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] [WIP] Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)