[13:35:57] <drdee>	 mooornig
[13:48:25] <ottomata>	 mooornign
[14:02:51] <drdee>	 yo qchris_away
[14:10:56] <qchris>	 Hi drdee.
[14:11:14] <drdee>	 good afternoon!
[14:11:22] <qchris>	 Good morning :-)
[14:25:36] <drdee>	 qchris, bat cave to talk about geo wiki?
[14:25:50] <qchris>	 Ok. Booting the google machine.
[14:27:49] <drdee>	 thanks
[15:03:43] <milimetric>	 hi! sorry for the late arrival
[15:03:49] <milimetric>	 anyone see average today?
[15:04:12] <drdee>	 no
[15:04:17] <drdee>	 he is not around
[15:13:41] <grrrit-wm>	 (PS1) Ottomata: Using tempfile script for large Hive add partition queries. [analytics/kraken] - https://gerrit.wikimedia.org/r/90553
[15:16:08] <ottomata>	 milimetric: cool? ^
[15:16:30] <milimetric>	 checking
[15:18:06] <milimetric>	 mmm, I think that logic would be better inside self.query
[15:18:21] <milimetric>	 you could just check len(query) and if it's big, use the tempfile
[15:18:58] <milimetric>	 that way anything calling query would get that benefit
[15:19:03] <milimetric>	 ottomata: ^
[15:19:11] <milimetric>	 adding to gerrit (duh, sorry :))
[15:19:59] <grrrit-wm>	 (CR) Milimetric: "(1 comment)" [analytics/kraken] - https://gerrit.wikimedia.org/r/90553 (owner: Ottomata)
[15:20:07] <ottomata>	 hmmm
[15:20:17] <ottomata>	 not bad idea i think but maybe it isn't obvious to users of query?
[15:20:18] <ottomata>	 hm
[15:20:20] <ottomata>	 yeah maybe...
[15:20:33] <ottomata>	 the error is actually from subprocess
[15:20:36] <ottomata>	 too many arguments
[15:20:41] <ottomata>	 because it takes a list
[15:20:44] <ottomata>	 instead of a long string
[15:21:36] <milimetric>	 then maybe just always do it using a tempfile?
[15:21:45] <ottomata>	 hmm
[15:21:59] <ottomata>	 yeah maybe so.
[15:22:00] <ottomata>	 ok
[15:22:58] <ottomata>	 hmm, but that means all of those show create table and show partitions statements etc. will use tempfiles too
[15:23:22] <milimetric>	 oh, instead of self.command, why not just use sh?
[15:23:23] <CommanderData>	 That wouldn't be economically feasible
[15:24:29] <milimetric>	 oh duh, that's what command is
[15:24:31] <ottomata>	 yeah
[15:24:38] <ottomata>	 ummmmm, yeah i don't like always using temp
[15:24:38] <ottomata>	 oit works
[15:24:42] <ottomata>	 but it makes —verbose more useless
[15:24:47] <ottomata>	 because you don't see what queries are running
[15:24:54] <ottomata>	 i could log from the query method I suppose...
[15:24:54] <ottomata>	 hm
[15:25:10] <milimetric>	 so wait this throws an error?
[15:25:19] <milimetric>	 when there are too many partitions?
[15:25:22] <ottomata>	 yes
[15:25:24] <milimetric>	 and the magic number is 20
[15:25:29] <ottomata>	 on
[15:25:30] <ottomata>	 no
[15:25:32] <ottomata>	 i just picked 20
[15:25:46] <ottomata>	 but, i ran this for our imports
[15:25:48] <milimetric>	 that seems really strange that it sometimes throws errors then, what's "too many"
[15:25:51] <ottomata>	 and there were like 2K some partitions
[15:26:26] <ottomata>	 it could be a shell thing, dunno
[15:27:06] <milimetric>	 ok, I know we don't want to waste too much time on this, but the right thing to do seems to me:
[15:27:10] <milimetric>	 1. figure out why it fails
[15:27:11] <CommanderData>	 Our code quality is no worse than anyone else in the industry
[15:27:25] <milimetric>	 2. in self.query, check for that and use the tempfile trick to avoid the error
[15:28:29] <milimetric>	 I can try to help, but I don't think I have access to run that hive command
[15:31:07] <ottomata>	 i think it lets anyone run it, which is kinda weird
[15:31:11] <ottomata>	 but i've already create the partitions anyway
[15:31:16] <ottomata>	 soooo finding it hard to reproduce now
[15:31:18] <ottomata>	 i'd have to mock it
[15:31:26] <ottomata>	 i'm playing with subprocess
[15:31:38] <ottomata>	 milimetric:
[15:31:47] <ottomata>	 how about I just make add partitions always use tempfile?
[15:32:01] <ottomata>	 i could make force tempfile an arg to query?
[15:33:09] <milimetric>	 sure, that's a fine compromise for now
[15:39:35] <grrrit-wm>	 (PS2) Ottomata: Using tempfile script for Hive add partition queries. [analytics/kraken] - https://gerrit.wikimedia.org/r/90553
[15:39:38] <ottomata>	 milimetric: ^
[15:40:46] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Using tempfile script for Hive add partition queries. [analytics/kraken] - https://gerrit.wikimedia.org/r/90553 (owner: Ottomata)
[15:40:53] <milimetric>	 looks good
[15:41:42] <ottomata>	 k deploying and going to setup a cron to just run these once an hour
[15:48:56] <travis-ci>	 [travis-ci] develop/ec09691 (#144 by milimetric): The build has errored. http://travis-ci.org/wikimedia/limn/builds/12723764
[15:50:14] <milimetric>	 once an hour makes sense ottomata, did you set up the importer to run nightly?
[15:51:24] <ottomata>	 no haven't automated it all yet
[16:13:23] <drdee>	 @rss+ mingle https://mingle.corp.wikimedia.org/projects/analytics/feeds.atom?filter_types[cards]=Card::Version&filter_types[pages]=Page::Version
[16:13:24] <wm-bot>	 Item was inserted to feed
[16:13:59] <drdee>	 @rss-on
[16:14:00] <wm-bot>	 Rss feed is enabled on this channel
[16:15:00] <wm-bot>	 [mingle] Defect #1213 [Bug 55878] Top 40 pixels of limn pages is blocked by something invisible changed by Diederik van Liere: Release Schedule - Release set to #803 FY 2013 Quarter 2: October 3 - December 31,Release Schedul... https://mingle.corp.wikimedia.org/projects/analytics/cards/1213?version=8
[16:15:00] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1213
[16:15:01] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/803
[16:15:14] <wm-bot>	 [mingle] Defect #1213 [Bug 55878] Top 40 pixels of limn pages is blocked by something invisible changed by Diederik van Liere: Epic Release set to #803 FY 2013 Quarter 2: October 3 - December 31,Epic Release - Epic set to #8... https://mingle.corp.wikimedia.org/projects/analytics/cards/1213?version=9
[16:15:15] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1213
[16:15:15] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/803
[16:15:15] <wm-bot>	 [mingle] Defect #1213 [Bug 55878] Top 40 pixels of limn pages is blocked by something invisible changed by Diederik van Liere: Estimate set to 1 https://mingle.corp.wikimedia.org/projects/analytics/cards/1213?version=10
[16:15:15] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1213
[16:15:29] <wm-bot>	 [mingle] Defect #1213 [Bug 55878] Top 40 pixels of limn pages is blocked by something invisible changed by Diederik van Liere: Responsible set to Dan Andreescu https://mingle.corp.wikimedia.org/projects/analytics/cards/1213?version=11
[16:15:30] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1213
[16:15:30] <wm-bot>	 [mingle] Defect #1213 [Bug 55878] Top 40 pixels of limn pages is blocked by something invisible changed by Diederik van Liere: Development Status changed from Backlog to Done https://mingle.corp.wikimedia.org/projects/analytics/cards/1213?version=12
[16:15:30] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1213
[16:15:31] <drdee>	 okay it seems to work :)
[16:15:44] <drdee>	 let's dial it back a little bit
[16:16:23] <drdee>	 first let's shut up CommanderData
[16:22:23] <wm-bot>	 [mingle] Defect #1213 [Bug 55878] Top 40 pixels of limn pages is blocked by something invisible changed by Diederik van Liere: Class of Service changed from Standard to Expedited https://mingle.corp.wikimedia.org/projects/analytics/cards/1213?version=13
[16:22:23] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1213
[16:48:40] <qchris>	 So wm-bot does eat atom feeds after all? Nice!
[16:56:23] <ottomata>	 milimetric: mind if I drop the test tables in the default database?
[16:56:35] <milimetric>	 not at all
[16:56:47] <milimetric>	 DarTar: ottomata is dropping milimetric_pagecounts and milimetric_pagecounts_daily
[16:56:55] <milimetric>	 we're creating real tables in their place
[16:57:02] <ottomata>	 but that's because there is now a real pagecounts table
[16:57:04] <ottomata>	 in the wmf databsae
[16:57:05] <DarTar>	 great, thx for the heads up
[16:57:06] <ottomata>	 use wmf;
[16:57:07] <ottomata>	 show tables;
[16:57:14] <ottomata>	 with data since august!
[16:57:17] <milimetric>	 :)
[16:57:19] <DarTar>	 nice
[16:58:32] <Ironholds>	 'the wmf database'/
[16:58:36] <Ironholds>	 *?
[16:58:51] <ottomata>	 yeah, rather than one off user databases
[16:58:59] <ottomata>	 if you want to create tables from the raw imported data tables yourself
[16:59:06] <ottomata>	 you should create a database with your username
[16:59:09] <ottomata>	 and create them there
[16:59:21] <Ironholds>	 ahh, gotcha
[17:01:51] <wm-bot>	 [mingle] Feature #699 Measure milestones achieved by an editor (threshold) changed by Diederik van Liere: Development Status changed from Showcasing to Sign-off https://mingle.corp.wikimedia.org/projects/analytics/cards/699?version=52
[17:01:51] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/699
[17:03:34] <drdee>	 DarTar can you look at #699
[17:03:34] <CommanderData>	 drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/699
[17:05:21] <wm-bot>	 [mingle] Feature #701 Measure survival of an editor changed by Dan Andreescu: Development Status changed from Done to Coding and Testing https://mingle.corp.wikimedia.org/projects/analytics/cards/701?version=58
[17:05:22] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/701
[17:05:22] <wm-bot>	 [mingle] Feature #701 Measure survival of an editor changed by Dan Andreescu:  https://mingle.corp.wikimedia.org/projects/analytics/cards/701?version=59
[17:05:22] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/701
[17:05:36] <wm-bot>	 [mingle] Feature #701 Measure survival of an editor changed by Dan Andreescu: Release Schedule - Sprint changed from #802 2013/14 Q2 - Sprint 01 - WE 2013-10-16 to #1158 2013/... https://mingle.corp.wikimedia.org/projects/analytics/cards/701?version=59
[17:05:36] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/701
[17:05:36] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/802
[17:05:36] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1158
[17:06:29] <DarTar>	 drdee: I started the QA yesterday but haven't replied yet,
[17:06:50] <drdee>	 ty
[17:07:14] <DarTar>	 I'm a bit worried that this implements a sunset-less survival (i.e. activity *to date*), based on average's demo
[17:07:25] <DarTar>	 but I need to double check
[17:16:49] <DarTar>	 drdee: how do I see the revision history of a card in mingle?
[17:19:44] <drdee>	 there is a history link on the right side of the screen
[17:19:58] <drdee>	 if you are looking at a card
[17:20:20] <DarTar>	 that's not for a single card, it's for the whole board, right?
[17:20:34] <DarTar>	 oh i see
[17:20:36] <DarTar>	 scrap that
[17:20:38] <DarTar>	 thx
[17:22:15] <DarTar>	 I see the acceptance criteria were added by stefan, but they seem to conflict with the description of the card
[17:23:03] <DarTar>	 I'll send a detailed QA by mail anyway
[17:55:27] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=91
[17:55:27] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[17:55:56] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=92
[17:55:56] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[17:59:20] <ottomata>	 milimetric: before I go, i want to run this query with the seqs
[17:59:30] <milimetric>	 oh my code was fine
[17:59:34] <ottomata>	 http://codeshare.io/aITwc
[17:59:47] <ottomata>	 but i need to group by hostname i think
[17:59:48] <milimetric>	 right, ok
[18:01:53] <ottomata>	 let's get the count too
[18:01:58] <ottomata>	 so we can see the % loss
[18:02:27] <ottomata>	 yeah?
[18:02:28] <ottomata>	 try it?
[18:05:27] <ottomata>	 ha
[18:07:51] <milimetric>	 it no work?
[18:08:32] <ottomata>	 no
[18:08:49] <ottomata>	 FAILED: ParseException line 17:0 mismatched input '<EOF>' expecting Identifier near ')' in subquery source
[18:10:59] <milimetric>	 oh, I didn't see you type this - got time for hangout before you starve?  or, better, can I run this somewhere?
[18:11:00] <milimetric>	 you go eat
[18:11:30] <ottomata>	 yeah
[18:11:47] <ottomata>	 in batcave
[18:11:49] <ottomata>	 milimetric:
[18:42:03] <milimetric>	 hive metastore schema: https://issues.apache.org/jira/secure/attachment/12471108/HiveMetaStore.pdf
[19:04:44] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=93
[19:04:44] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:05:42] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=94
[19:05:42] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:06:41] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=95
[19:06:41] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:07:25] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=96
[19:07:25] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:08:10] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=97
[19:08:10] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:08:40] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=98
[19:08:40] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:10:38] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=99
[19:10:39] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:11:38] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=100
[19:11:39] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:14:11] <ottomata>	 milimetric:  good news, when I take out those dumb 0s (they are still messing hive partitioner up on some weird way), 0 loss!
[19:14:16] <ottomata>	 running to a cafe, back in a bit
[19:14:29] <milimetric>	 cool!
[19:15:26] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Bingle User: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=101
[19:15:27] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:28:52] <wm-bot>	 [mingle] Feature #1112 Gingle testcard changed by Diederik van Liere: Description changed https://mingle.corp.wikimedia.org/projects/analytics/cards/1112?version=102
[19:28:52] <CommanderData>	 wm-bot2 hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/1112
[19:40:47] <grrrit-wm>	 (PS1) Milimetric: fix incremental importing [analytics/kraken] - https://gerrit.wikimedia.org/r/90634
[19:41:06] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] fix incremental importing [analytics/kraken] - https://gerrit.wikimedia.org/r/90634 (owner: Milimetric)
[19:47:23] <kenanwang>	 hi I had an issue uploading a cohort to user metrics yesterday
[19:48:54] <milimetric>	 user metrics or wikimetrics?
[19:49:33] <kenanwang>	 user metrics
[19:49:57] <milimetric>	 hm...
[19:50:01] <milimetric>	 well, what was the issue
[19:50:18] <milimetric>	 we're not technically supporting that, but I don't want to like completely ignore you :)
[19:52:02] <milimetric>	 drdee, I see the problem now:
[19:52:11] <milimetric>	 https://mingle.corp.wikimedia.org/projects/analytics/cards/701 lists both "survivor" and "survived"
[19:52:18] <drdee>	 welp
[19:52:18] <milimetric>	 so I can see why it was confusing
[19:52:19] <CommanderData>	 It's a browser compatibility issue
[19:52:29] <milimetric>	 yes CommanderData, browsers are the devil
[19:52:38] <milimetric>	 anyway, renaming now
[19:52:43] <milimetric>	 I just want to confirm with you drdee
[19:52:54] <milimetric>	 "Survival" is the name of the metric and what shows up in the UI
[19:53:05] <milimetric>	 and "survived" is the label of the metric result in a report
[19:53:18] <drdee>	 that's my understanding as well, but let's check with DarTar
[19:53:22] <drdee>	 DarTar: ^^
[19:53:38] <milimetric>	 k
[20:00:18] <milimetric>	 hey drdee, kenanwang was asking a question about user metrics but he dc-ed.  Did you talk with him?
[20:00:27] <drdee>	 no
[20:00:30] <milimetric>	 we should follow up or at least point him to Dario if they haven't talked
[20:02:56] <DarTar>	 hey folks
[20:03:10] <DarTar>	 sorry I was afk, so:
[20:03:23] <DarTar>	 survival: metric name, survived: label
[20:04:20] <milimetric>	 yep
[20:04:24] <milimetric>	 renaming
[20:04:30] <milimetric>	 it was confusing because the card said "survivor" as well
[20:04:32] <DarTar>	 kenan: I talked to him and in fact shared with my my usermetrics creds (the ones he got from ryan or drdee don't work - the pw is not encrypted in the same way as the other ones in the user table)
[20:04:46] <milimetric>	 oh
[20:04:48] <milimetric>	 ok
[20:05:08] <DarTar>	 but we're hitting a serious problem of people not being able to use wikimetrics and relying on me for access to an unmaintained usermetrics
[20:05:35] <drdee>	 what do you mean by "not being able to use wikimetrics " ?
[20:05:54] <DarTar>	 cohorts of thousands of users
[20:05:55] <DarTar>	 or 10ks
[20:05:56] <milimetric>	 because of feature parity
[20:05:58] <milimetric>	 right
[20:06:40] <drdee>	 you lowered priority on wikimetrics in favor of the page view api :)
[20:06:46] <DarTar>	 so like I said, I think within Wikimetrics stories we should prioritize the scalability concerns over other functionality
[20:06:54] <drdee>	 we are working on 818
[20:07:05] <DarTar>	 I know, but that was under the assumption that people wouldn't use it
[20:07:11] <DarTar>	 and that's fine
[20:07:20] <drdee>	 #818
[20:07:20] <CommanderData>	 drdee hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/818
[20:07:42] <DarTar>	 yes 818 would be a big thing
[20:07:52] <drdee>	 it's in progres
[20:08:01] <milimetric>	 yeah, DarTar, I'm on it :)
[20:08:13] <milimetric>	 I'm gonna "knock it out of the park" drdee said :)
[20:08:19] <milimetric>	 no pressure :P
[20:08:32] <Ironholds>	 okay, now starting my assigned 'break hadoop and tell us how you did it' day
[20:08:39] <milimetric>	 woo!
[20:08:45] <Ironholds>	 well, sorta-woo
[20:08:48] <milimetric>	 sadly, I'm about to leave in an hour or so
[20:08:52] <Ironholds>	 I may or may not have forgotten how to actually access hadoop.
[20:08:56] <milimetric>	 nono, I stand by my \o/
[20:09:01] <Ironholds>	 I'm sat on analytics 1001 feeling very silly
[20:09:21] <milimetric>	 you should try analytics1011
[20:09:43] <milimetric>	 or ottomata said we might want to start working on 1027 but I'm not sure if that's ready yet
[20:10:40] <Ironholds>	 ech, I'm on 1011, just misremembering. bah.
[20:11:00] <milimetric>	 k, so why you feeling silly
[20:11:01] <CommanderData>	 I must not have understood what you were asking for
[20:11:17] <milimetric>	 (disregard the impolite robot)
[20:11:55] <ottomata>	 analytics1026*
[20:12:04] * milimetric  just realized that Oliver's initials are OK

[20:12:11] <ottomata>	 Ironholds:  I'm here for a bit
[20:12:13] <ottomata>	 can help!
[20:12:28] <Ironholds>	 milimetric: yeah, yeah ;p
[20:12:34] <ottomata>	 but an11 is fine for now
[20:12:50] <Ironholds>	 ottomata: oh, it's the dumbest thing: I've forgotten the command to actually jump into hive to throw queries around
[20:12:55] <Ironholds>	 everything else? perfectly set up in my head.
[20:13:07] <DarTar>	 hive ?
[20:13:10] <DarTar>	 :)
[20:13:24] <milimetric>	 "hive"
[20:13:28] <Ironholds>	 DarTar: I hate and love you in equal amounts
[20:13:48] <DarTar>	 Ironholds: do you have access to the bookshelf on dropbox ?
[20:14:07] <Ironholds>	 nope, but I've sat myself down with Prokopp's book
[20:14:29] <milimetric>	 in general, Ironholds, they really tried to mimic the mysql world as closely as possible
[20:14:44] <Ironholds>	 milimetric: only with less suckage, it seems
[20:14:44] <ottomata>	 ha
[20:14:50] <Ironholds>	 error messages that actually report where the error is!
[20:14:56] <Ironholds>	 it's glorious.
[20:15:17] <milimetric>	 heh, ok, you wait 'till you get the "could not convert integer to double" error
[20:15:26] <milimetric>	 which basically is as useful as a check engine light on a car
[20:15:43] <Ironholds>	 yeah, but it beats "you have an error on line 1" when the error is actually on, say, line 8
[20:15:54] <Ironholds>	 which is pretty common if you select foo AS bar and then reference it as foo later on.
[20:16:03] <Ironholds>	 okay, noms, then breaking stuff. back in a tick.
[20:16:13] <milimetric>	 it is pretty good, I gotta admit.  Especially when you try to do stuff you're used to and you have to learn something new about hive.  It's good at that hand holding
[20:16:59] <ottomata>	 milimetric: looks like pagecounts has the same problem
[20:17:04] <ottomata>	 i'll have to recreate all those partitions
[20:17:13] <milimetric>	 k
[20:20:56] <wm-bot>	 [mingle] Page Requests Grantmaking Learning & Evaluation created by Diederik van Liere: Content changed https://mingle.corp.wikimedia.org/projects/analytics/wiki/Requests_Grantmaking_Learning_&amp;_Evaluation?version=1
[20:21:27] <wm-bot>	 [mingle] Page Customers changed by Diederik van Liere: Content changed https://mingle.corp.wikimedia.org/projects/analytics/wiki/Customers?version=13
[20:24:11] <wm-bot>	 [mingle] Page Requests Program Evaluation and Design created by Diederik van Liere:  https://mingle.corp.wikimedia.org/projects/analytics/wiki/Requests_Program_Evaluation_and_Design?version=1
[20:24:27] <wm-bot>	 [mingle] Page Requests Program Evaluation and Design created by Diederik van Liere: Content changed https://mingle.corp.wikimedia.org/projects/analytics/wiki/Requests_Program_Evaluation_and_Design?version=1
[20:24:57] <wm-bot>	 [mingle] Page Customers changed by Diederik van Liere: Content changed https://mingle.corp.wikimedia.org/projects/analytics/wiki/Customers?version=14
[20:28:50] <grrrit-wm>	 (PS1) Ottomata: Separating out ADD PARTITION statements into individual queries. [analytics/kraken] - https://gerrit.wikimedia.org/r/90646
[20:29:00] <grrrit-wm>	 (PS2) Ottomata: Separating out ADD PARTITION statements into individual queries. [analytics/kraken] - https://gerrit.wikimedia.org/r/90646
[20:29:13] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Separating out ADD PARTITION statements into individual queries. [analytics/kraken] - https://gerrit.wikimedia.org/r/90646 (owner: Ottomata)
[20:30:03] <grrrit-wm>	 (PS1) Milimetric: cleaning up the Survival name and language [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/90648
[20:30:16] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] cleaning up the Survival name and language [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/90648 (owner: Milimetric)
[20:32:14] <milimetric>	 DarTar: cards #701 and #699 are in sign-off for your review
[20:32:14] <CommanderData>	 milimetric hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/701
[20:32:14] <CommanderData>	 milimetric hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/699
[20:32:29] <wm-bot>	 [mingle] Feature #701 Measure survival of an editor changed by Dan Andreescu: Development Status changed from Coding and Testing to Sign-off https://mingle.corp.wikimedia.org/projects/analytics/cards/701?version=60
[20:32:43] <milimetric>	 please take a look and let us know if there's anything else wrong
[20:37:29] <ottomata>	 ok, milimetric pagecounts partitions should be back
[20:38:04] <milimetric>	 awesome.  So DarTar ^ that means table "pagecounts" in database "wmf" has all the hourly data from August 1st to now
[20:38:13] <milimetric>	 and IH|noms, same ^
[20:39:57] <DarTar>	 that's great :)
[20:50:20] <Ironholds>	 milimetric: yep, saw. thanks!
[20:55:58] <wm-bot>	 [mingle] Feature #818 Asynchronous cohort validation changed by Dan Andreescu: Development Status changed from Ready for Dev to Coding and Testing https://mingle.corp.wikimedia.org/projects/analytics/cards/818?version=17
[21:02:13] <DarTar>	 milimetric: is there any known issue in wikimetrics with cohort creation by user ID (as opposed to username) at the moment?
[21:02:24] <milimetric>	 not that i know of, no
[21:02:28] <milimetric>	 why, what's the issue?
[21:02:29] <CommanderData>	 Our redundant systems must have failed as well
[21:02:50] <DarTar>	 trying to upload a small cohort of new registered users
[21:02:56] <DarTar>	 using only user_ids
[21:03:07] <milimetric>	 can you send it?  I'll take a look
[21:03:18] <DarTar>	 no project specified (selecting a default project from the UI)
[21:03:26] <milimetric>	 yeah, that should all work
[21:03:37] <drdee>	 milimetric: trying to run nosetests get error "ConnectionError: Error 61 connecting localhost:6379. Connection refused."
[21:03:37] <DarTar>	 k sending you the list
[21:03:53] <milimetric>	 you don't have redis installed drdee
[21:03:58] <drdee>	 k
[21:04:03] <milimetric>	 but run scripts/test
[21:04:04] <drdee>	 not running you mean ;)
[21:04:08] <milimetric>	 it's more thorough and deletes stuff
[21:04:11] <milimetric>	 right
[21:08:34] <average>	 guys, good luck with #818 .
[21:08:34] <CommanderData>	 average hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/818
[21:08:50] <average>	 I saw Dan wrote earlier that he's working on the card.
[21:09:13] <milimetric>	 hey Stefan
[21:09:19] <DarTar>	 milimetric: mail for you
[21:09:20] <average>	 hey Dan
[21:09:21] <milimetric>	 I'm just writing some tests at the moment
[21:09:22] <DarTar>	 hi average
[21:09:30] <average>	 hi Dario
[21:09:40] <milimetric>	 saw it DarTar, testing now
[21:09:51] <DarTar>	 thx
[21:10:09] <milimetric>	 I'm not taking over the card completely average, I just think it needs to get done sooner than we thought so it would be good to work together on it
[21:10:20] <milimetric>	 i'll submit anything I do to gerrit and rely on you to merge
[21:10:43] <average>	 I think we agreed that I would work on the card
[21:10:55] <average>	 I think at this moment there's no point in disagreeing
[21:11:57] <milimetric>	 ah, DarTar, it appears our problem is one of latency
[21:12:08] <milimetric>	 so the users you are trying to upload don't exist in labsdb yet
[21:12:09] <DarTar>	 oh, replag?
[21:12:11] <milimetric>	 19948581
[21:12:13] <DarTar>	 ha
[21:12:15] <milimetric>	 that's the max user_id
[21:12:20] <DarTar>	 good point, hadn't thought of that
[21:12:25] <DarTar>	 they're from the last hour indeed
[21:12:28] <milimetric>	 so yours are still about 10k users away
[21:12:36] <milimetric>	 that's interesting though
[21:12:46] <milimetric>	 so we've got a lag of roughly 10k users, do you know how many hours that is?
[21:12:47] <DarTar>	 yeah, I am also pulling these from a slave
[21:12:56] <DarTar>	 let me check
[21:13:03] <DarTar>	 10K is 2 days
[21:13:12] <DarTar>	 for enwiki
[21:13:43] <milimetric>	 that last user registered yesterday around 2pm
[21:13:44] <milimetric>	 hm.....
[21:14:40] <milimetric>	 that's kind of a long time to wait for analysis if you're trying to look at very recent data
[21:15:11] <milimetric>	 doesn't help to find out 1/2 our editors deleted their accounts 24 hours after a bad feature was introduced :)
[21:15:23] <average>	 I just ended my contract on oDesk
[21:15:27] <average>	 good luck with #818 guys
[21:15:27] <CommanderData>	 average hopes that someone will have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/818
[21:15:28] <average>	 have fun
[21:15:41] <milimetric>	 ?
[21:16:06] <drdee>	 ??
[22:10:15] <milimetric>	 have a good weekend everyone.
[22:12:31] <drdee>	 laterz
[22:32:28] <yurik_>	 is drdee around?
[22:34:27] <yurik_>	 or is DarTar the same as drdee?
[22:35:26] <yurik_>	 in any case - i might have found another reason for analytics mismatch for zero - https://gerrit.wikimedia.org/r/#/c/90665/
[22:36:03] <DarTar>	 no DarTar != drdee :p
[22:38:51] <Ironholds>	 anyone got any objection to me using the test db as a, well, test db?
[22:44:01] <mwalker>	 somewhat interesting analytics question; does anyone know how many requests we serve that are uncompressed?
[23:50:43] <DarTar>	 Ironholds: I'm pretty sure you can, but all the devs are off
[23:51:39] <DarTar>	 ottomata had to fix a permission problem for my user, not sure if that applies to you too
[23:52:40] <DarTar>	 mwalker: interesting, but I have no clue - that's probably a question for drdee or qchris
[23:53:20] <mwalker>	 *nods*
[23:53:59] <qchris>	 mwalker: Let me check whether we have that data available somewhere ... although I doubt it ...
[23:55:25] <qchris>	 mwalker: As I assumed, I cannot find this data in the logs that we hold.
[23:56:05] <mwalker>	 yep -- so my plan to collect it is to use https://gerrit.wikimedia.org/r/#/c/90667/ and compare the outbound object sizes
[23:56:13] <qchris>	 But as the more experienced guys are off into the weekend already, I think it's best to bring it up on the mailing list
[23:56:23] <qchris>	 Ah... me looks.
[23:56:33] <mwalker>	 seems reasonable to bring it up on the list
[23:57:06] <mwalker>	 but ya; basically since I know the expected sizes of content sent down; I should be able to tell if things are sent compressed or not
[23:59:20] <qchris>	 Do the logs contained the compressed size?
[23:59:31] <qchris>	 I totally do not know ... I never looked at that.
[23:59:40] <mwalker>	 not entirely sure to be honest -- I would assume so