[00:56:05] <Ironholds>	 hey, analytics engineering!
[00:56:10] <Ironholds>	 we're performing a stress test on hive
[00:56:14] <Ironholds>	 that'll be $2,000
[05:51:29] <wikibugs>	 Analytics / General/Unknown: datasets.wikimedia.org SSL error - https://bugzilla.wikimedia.org/72805 (Dario Taraborelli) NEW p:Unprio s:normal a:None https://datasets.wikimedia.org/public-datasets gives a 404 (meaning that people on SSL can't access any data available at this URL). See http...
[12:51:14] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian)
[12:51:14] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful due to only bits caches causing unknown problems - https://bugzilla.wikimedia.org/72809 (christian) NEW p:Unprio s:normal a:None In this bug, we track issues around raw webrequest partitions (not) being marked s...
[12:52:13] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435 (christian)
[12:52:13] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful due to only bits caches causing unknown problems - https://bugzilla.wikimedia.org/72809 (christian)
[12:52:13] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful - https://bugzilla.wikimedia.org/70085 (christian)
[12:52:43] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful due to only bits caches causing unknown problems - https://bugzilla.wikimedia.org/72809 (christian)
[12:52:43] <wikibugs>	 Analytics / Refinery: Several raw webrequest partitions now marked successful between 2014-10-13T13:xx:xx and 2014-10-13T22:xx:xx - https://bugzilla.wikimedia.org/72028 (christian)
[12:52:44] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435 (christian)
[12:52:57] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions for 2014-10-07T1[789]:xx:xx not marked successful - https://bugzilla.wikimedia.org/71882 (christian)
[12:53:12] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435 (christian)
[12:53:13] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful due to only bits caches causing unknown problems - https://bugzilla.wikimedia.org/72809 (christian)
[12:53:28] <wikibugs>	 Analytics / Refinery: Duplicates/missing logs from esams bits for 2014-09-28T{18,19,20}:xx:xx - https://bugzilla.wikimedia.org/71435 (christian)
[12:53:28] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful due to only bits caches causing unknown problems - https://bugzilla.wikimedia.org/72809 (christian)
[12:53:29] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions for 2014-10-08T1[89]:xx:xx not marked successful - https://bugzilla.wikimedia.org/71881 (christian)
[12:54:58] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions for 2014-10-30T21/1H not marked successful - https://bugzilla.wikimedia.org/72810 (christian) NEW p:Unprio s:normal a:None The bits and upload webrequest partition [1] for 2014-10-30T21/1H have not been marked successful.  What happened?   [1]...
[12:55:12] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions for 2014-10-30T21/1H not marked successful - https://bugzilla.wikimedia.org/72810#c1 (christian) For bits, it only affected cp3020. The affected period is 2014-10-30T21:25:41/2014-10-30T21:26:26. No lost messages, only 70660 duplicates, which is <2 second...
[12:55:13] <wikibugs>	 Analytics / Refinery: Raw webrequest partitions that were not marked successful due to only bits caches causing unknown problems - https://bugzilla.wikimedia.org/72809 (christian)
[12:57:09] <qchris>	 !log Marked raw upload webrequest partition for 2014-10-30T21/1H ok (See {{bug|72810}})
[12:57:13] <qchris>	 !log Marked raw bits webrequest partition for 2014-10-30T21/1H ok (See {{bug|72810}})
[13:34:58] <wikibugs>	 Analytics / Refinery: Kafka logs drowning in errors processing fetch requests since 2014-10-28 ~20:00 - https://bugzilla.wikimedia.org/72812 (christian) NEW p:Unprio s:normal a:None Since 2014-10-28 ~20:00, the kafka logs on each four brokers are drowning [1] in exceptions around processing...
[13:35:41] <ottomata>	 uhhh
[13:35:45] <ottomata>	 drowning!?
[13:41:42] <wikibugs>	 Analytics / Refinery: analytics1021 getting kicked out of kafka partition leader role on 2014-10-27 ~07:12 - https://bugzilla.wikimedia.org/72550 (christian)
[13:41:43] <wikibugs>	 Analytics / Refinery: Kafka logs drowning in kafka.server.KafkaApis errors processing fetch requests since 2014-10-28 ~20:00 - https://bugzilla.wikimedia.org/72812#c1 (christian) The time 2014-10-28 ~20:00 nicely matches the leader re-election of bug 72550 comment 1.
[13:42:44] <wikibugs>	 Analytics / Refinery: analytics1021 getting kicked out of kafka partition leader role on 2014-10-27 ~07:12 - https://bugzilla.wikimedia.org/72550 (christian)
[13:42:44] <wikibugs>	 Analytics / Refinery: Kafka logs drowning in kafka.server.KafkaApis errors processing fetch requests since 2014-10-28 ~20:00 - https://bugzilla.wikimedia.org/72812 (christian)
[14:17:25] <ottomata>	 qchris_away: did you say you are leaving for the weekend?
[14:17:31] <ottomata>	 sooN?
[14:17:37] <qchris_away>	 ottomata: actually ... right now :-)
[14:17:53] <qchris_away>	 Do you need me around?
[14:18:13] <ottomata>	 naw, its ok, i just like saying things like "LOOK everything is FINE" when you are around
[14:18:20] <ottomata>	 because then you can say "wait, no, look, it is not fine here"
[14:18:25] <ottomata>	 :)
[14:18:28] <qchris_away>	 Hahahaha :-)
[14:18:52] <qchris_away>	 I'll go through the channel backlog tomorrow ;-)
[14:19:00] <ottomata>	 haa, ok
[14:19:12] <qchris_away>	 Is it about the tsv vetting, or something else?
[14:20:27] <ottomata>	 tsv vetting
[14:20:34] <ottomata>	 but, that can mostly wait til monday anyway
[14:20:37] <ottomata>	 so we have mroe data
[14:21:00] <qchris_away>	 For the tsv vetting, you were doing the hard work ... I was only sitting besides you :-)
[14:21:12] <ottomata>	 yeha, but 4 eyes are better than 2
[14:21:27] <qchris_away>	 I'll return in a few hours. Then I'll have a look at the tsvs. Ok?
[14:21:37] <ottomata>	 nonon
[14:21:39] <ottomata>	 go away!
[14:21:50] <ottomata>	 i'm just choosing what to work on today, this can wait!
[14:21:50] <qchris_away>	 Btw. ... kafkatee edit.tsv for 2014-10-30 and 2014-10-31 are missing on stat1002.
[14:22:01] <ottomata>	 yes, they were never synced
[14:22:05] <ottomata>	 i synced them manually the other day
[14:22:09] <qchris_away>	 Oh. I see.
[14:22:31] <ottomata>	 I will ahve a look at your kafka broker bugs first anyway
[14:22:47] <qchris_away>	 Thanks.
[14:23:04] <qchris_away>	 The brokers are working nicely and doing they work just fine.
[14:23:15] <qchris_away>	 Only the logs are just full of those errors :-(
[14:23:26] <qchris_away>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=72812
[14:23:38] <qchris_away>	 But that can probably wait a few days too.
[14:23:52] <qchris_away>	 Anyways .... enjoy your weekend :-)
[14:24:48] <ottomata>	 hmm, i wonder if this is part of ellery's problem, he might be doing it
[14:24:54] <ottomata>	 he is running kafkacat in a cron somewhere
[14:25:02] <ottomata>	 and things are not workgin right for him
[14:42:57] <wikibugs>	 Analytics / Refinery: Kafka logs drowning in kafka.server.KafkaApis errors processing fetch requests since 2014-10-28 ~20:00 - https://bugzilla.wikimedia.org/72812#c2 (Andrew Otto) NEW>RESO/WON Ah, this is caused by Ellery's use of kafkacat on stat1002.  It looks like kafkacat does not properly c...
[14:45:01] <milimetric>	 kevinator: mediawiki storage is the blocker for annotations
[14:45:08] <milimetric>	 (from your msg last night)
[14:45:37] <milimetric>	 once we have a place to put annotations, it's trivial to render them - UX is a really nice to have but we can render them ourselves too
[14:46:55] <milimetric>	 kevinator / nuria__ / mforns: btw, Sean is testing the warehouse schema / loading queries in a database called "warehouse" on analytics-store so he can get an idea of the load / etc.  A-w-e-s-o-m-e! :D
[14:47:31] <mforns>	 ;]
[14:54:17] <nuria__>	 milimetric: nice, real nice
[15:07:04] <mforns>	 milimetric, nuria__: I got the line where the exception is thrown: it's validate_cohort.py line 243
[15:07:13] <mforns>	 matches = session.query(MediawikiUser).filter(clause).all()
[15:07:52] <mforns>	 which is already within a try block, and in fact is the only query within that try block
[15:07:56] <nuria__>	 mforns: and the exception is which one?
[15:08:09] <mforns>	 OperationalError: (OperationalError) (2005, "Unknown MySQL server host 'wikimania2015wiki.labsdb' (0)") None None
[15:08:38] <mforns>	 should I catch this error separately and do nothing?
[15:10:36] <nuria__>	 mforns: i think we need to distinguish this case (recoverable)
[15:10:46] <nuria__>	 from another exception (not recoverable)
[15:10:53] <mforns>	 yes
[15:10:54] <mforns>	 ok
[15:10:55] <nuria__>	 this exception doesn't need to be re-raised
[15:11:06] <nuria__>	 so we need two catch blocks
[15:11:18] <mforns>	 yes, that's what I meant with 'doing nothing'
[15:11:21] <mforns>	 :]
[15:11:23] <nuria__>	 rather tahn 1 on line 272
[15:11:34] <mforns>	 understand
[15:12:05] <nuria__>	 one catch +plus log (without raise) for this exceptions, validation continues for the rest of the projects
[15:12:52] <nuria__>	 and we can have (gasp) a catch all , log and raise for anything but, but this should not be caught above, makes sense?
[15:13:39] <nuria__>	 also, likely this  # clear out the dictionary in case of an exception, and raise the exception 	280
[15:13:39] <nuria__>	 278	        for key in users_dict.keys():	        for key in users_dict.keys():	281
[15:13:39] <nuria__>	 279	            users_dict.pop(key)	            users_dict.pop(key)
[15:13:59] <nuria__>	 needs to happen for both exceptions
[15:14:19] <mforns>	 aha
[15:14:50] <nuria__>	 so we disntiguish the "recoverable" case (no need to re-raise) from the non recoverable
[15:15:14] <mforns>	 yes, got it
[15:15:23] <mforns>	 thanks!!
[15:15:30] <nuria__>	 callers do not need to catch anything on that case, exception gets handled where thrown
[15:15:33] <nuria__>	 k
[15:15:50] <nuria__>	  mforns , milimetric: taking my daughter to school, brb
[15:15:57] <mforns>	 ok
[15:23:18] <Ironholds>	 nuria__ has a kid?
[15:23:30] <Ironholds>	 would that be a...nuriette? nurietta?
[15:34:34] <milimetric>	 well, actually nuria's name is nurieta (not sure about the spelling)
[15:34:53] <milimetric>	 and derivative humans don't follow naming conventions
[15:35:52] <Ironholds>	 they should
[15:36:10] <Ironholds>	 and they do!
[15:36:26] <Ironholds>	 That's why we have Ben Foo or Bar Foo as a naming convention, or Fooson or Foosdottir
[15:36:44] <Ironholds>	 I'll just think of them as nurieta.child.class$new()
[15:59:10] <milimetric>	 :)
[16:00:28] <milimetric>	 random_mutation(nurieta.class.merge(nurieta.husband.class))()
[16:01:09] <milimetric>	 in case nuria comes back, we'll have to explain ourselves: no big deal, just trying to define your kid in python / R, nothing to see here :P
[16:01:33] <Ironholds>	 naw, you wouldn't use nurieta.child.class$new() in R
[16:01:42] <Ironholds>	 that's a mix of S3/4 and R5/6 syntax. 2-4 different class systems!
[16:02:02] <Ironholds>	 it'd end up with a nurieta method that applies to objects of the class "child.class". Nobody wants that.
[16:02:52] <Ironholds>	 you'd probably define a class in R6 that inherits from nurieta and nurieta_partner classes and then just initialize a new instance of that. So child_class <- setRefClass("child_class", inherits = c("nurieta_class","nurieta_partner_class")) and then class$new()
[16:26:18] <nuria__>	 pufff .. no comments
[17:14:41] <nuria__>	 ottomata: one question about eventlogging and puppet
[17:15:14] <ottomata>	 yesssaksmeee
[17:15:45] <nuria__>	 ottomata: as far as i can see new upgrades of packages are not handled via puppet
[17:16:00] <ottomata>	 usually not
[17:16:02] <nuria__>	 ottomata: so if we wanted to add a new python package
[17:16:07] <nuria__>	 1) stop service
[17:16:19] <nuria__>	 2) pip or apt-get (likely apt-get as this is prod)
[17:16:24] <nuria__>	 3) restart
[17:16:26] <ottomata>	 yup
[17:16:31] <ottomata>	 you can probalby jsut apt-get and then restart
[17:16:45] <ottomata>	 don't need to stop it really, as the process has the current stuff loaded in mem
[17:16:46] <nuria__>	 ottomata: and do you know....
[17:16:56] <nuria__>	 ottomata: how does new code get deployed to vanadium?
[17:17:06] <ottomata>	 new eventlogging code?
[17:17:07] <nuria__>	 ottomata: is it deployed to under the regular wmf schedule?
[17:17:09] <ottomata>	 i do not know
[17:17:12] <ottomata>	 i doubt it?
[17:17:27] <ottomata>	 ah
[17:17:30] <ottomata>	 it looks like via tin?
[17:17:32] <ottomata>	 git-deploy?
[17:17:32] <nuria__>	 ottomata: so it will be us sshing and git pull?
[17:17:38] <ottomata>	 hmmm
[17:17:53] <ottomata>	 no
[17:18:17] <nuria__>	 ottomata: good... cause that sounds a little too low tech
[17:21:03] <ottomata>	 yes, igt deploy
[17:21:09] <ottomata>	 nuria, deployment goes like this
[17:21:32] <ottomata>	 ssh tin.eqiad.wmnet
[17:21:32] <ottomata>	 cd /srv/deployment/eventlogging/EventLogging
[17:21:32] <ottomata>	 git pull
[17:21:32] <ottomata>	 git deploy start
[17:21:32] <ottomata>	 git deploy sync
[17:22:07] <ottomata>	 https://wikitech.wikimedia.org/wiki/Trebuchet#Deploying
[17:23:10] <nuria__>	 ottomata: k, i see, is there rollback? I see "abort" but that is no rollback
[17:23:24] <ottomata>	 sure
[17:23:30] <ottomata>	 git checkoiut HEAD^
[17:23:35] <ottomata>	 git deploy start; git deploy sync
[17:24:00] <nuria__>	 k, i see, redeploy
[17:24:23] <nuria__>	 ottomata: that assumes every check in into master is a deploy though
[17:24:41] <ottomata>	 git checkout <whatever_sha_you_want_)
[17:24:42] <ottomata>	 oh
[17:24:43] <ottomata>	 wait
[17:24:47] <ottomata>	 i think git deploy does tags
[17:24:51] <nuria__>	 ahhhh
[17:24:57] <nuria__>	 ottomata: that makes more sense
[17:25:11] <nuria__>	 cause you want to tag at deploy time to be able to rollback
[17:25:15] <nuria__>	 precisely
[17:25:28] <nuria__>	 to what you had before
[17:25:29] <ottomata>	 yup, each git-deploy start does a tag
[17:25:34] <nuria__>	 ahhh, k
[17:25:38] <ottomata>	 maybe sync too
[17:25:39] <ottomata>	 ?
[17:25:39] <nuria__>	 ottomata: that sounds better
[17:25:43] <ottomata>	 dunno exactly how that works
[17:25:43] <ottomata>	 but ja
[17:25:45] <ottomata>	 somethign like that
[17:26:22] <nuria__>	 ottomata: k, i am going to add a new dependency to EL so we will follow this procedure when we get there
[17:26:36] <ottomata>	 ok
[17:39:56] <wikibugs>	 Analytics / EventLogging: database consumer could batch inserts (sometimes) - https://bugzilla.wikimedia.org/67450#c3 (nuria) Actual beginning of e-mail thread with pertinent conversation: https://lists.wikimedia.org/pipermail/analytics/2014-August/002429.html
[17:42:40] <ottomata>	 brb, cafe time!
[18:17:43] <grrrit-wm>	 (PS14) Ottomata: Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142
[18:18:37] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[18:23:26] <grrrit-wm>	 (PS1) Ottomata: Updating refinery-hive/pom.xml's parent version to 0.0.2-SNAPSHOT to match rest of repo [analytics/refinery/source] - https://gerrit.wikimedia.org/r/170368
[18:23:41] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Updating refinery-hive/pom.xml's parent version to 0.0.2-SNAPSHOT to match rest of repo [analytics/refinery/source] - https://gerrit.wikimedia.org/r/170368 (owner: Ottomata)
[18:43:49] <grrrit-wm>	 (PS2) Mforns: Avoid exception accessing unknown project database [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/170152 (https://bugzilla.wikimedia.org/72582)
[18:48:21] <DarTar>	 milimetric: showing leila some ninja command-line parsing with jq, I forgot how awesome it is
[18:48:56] <DarTar>	 which reminds me I really need to order this new O’Reilly book
[18:49:24] <DarTar>	 http://datascienceatthecommandline.com/
[18:51:45] <grrrit-wm>	 (PS1) Ottomata: Release version 0.0.2 with UAParserUDF in refinery-hive [analytics/refinery/source] - https://gerrit.wikimedia.org/r/170373
[18:52:29] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Release version 0.0.2 with UAParserUDF in refinery-hive [analytics/refinery/source] - https://gerrit.wikimedia.org/r/170373 (owner: Ottomata)
[18:52:53] <milimetric>	 DarTar: jq?
[18:53:11] <DarTar>	 milimetric: awk for json
[18:53:13] <DarTar>	 http://stedolan.github.io/jq
[18:53:27] <DarTar>	 best thing since, well, awk
[18:56:38] <milimetric>	 s['props']['982']
[18:56:42] <milimetric>	 oops
[19:01:52] <milimetric>	 DarTar: cool! I never thought of having to hack around json files 'cause python and JS seem pretty up to the task
[19:02:03] <milimetric>	 but what's something I can do with jq in a more elegant way?
[19:02:26] <DarTar>	 single liners of hell, go pipes!
[19:03:14] <DarTar>	 grabbing lunch, brb
[19:04:12] <grrrit-wm>	 (PS1) Ottomata: Add refinery-hive-0.0.2 and refinery-tools-0.0.2 [analytics/refinery] - https://gerrit.wikimedia.org/r/170375
[19:05:02] <milimetric>	 ha :) I was just playing with https://jqplay.org/# and it seems pretty fun
[19:05:39] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add refinery-hive-0.0.2 and refinery-tools-0.0.2 [analytics/refinery] - https://gerrit.wikimedia.org/r/170375 (owner: Ottomata)
[19:07:04] <ottomata>	 oo, didn't know about jqplay, that's nice
[19:18:43] <grrrit-wm>	 (PS1) Ottomata: Bump version number after release [analytics/refinery/source] - https://gerrit.wikimedia.org/r/170380
[19:18:53] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Bump version number after release [analytics/refinery/source] - https://gerrit.wikimedia.org/r/170380 (owner: Ottomata)
[19:20:09] <grrrit-wm>	 (CR) Mforns: "Sorry for the late review" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/168202 (https://bugzilla.wikimedia.org/71582) (owner: Bmansurov)
[19:42:26] <ottomata>	 nuria__: at least I got the jar in the auxpath automatically
[19:42:37] <ottomata>	 now you just have to create temporary function uaparse as  "org.wikimedia.analytics.refinery.hive.UAParserUDF";
[19:42:39] <ottomata>	 no need to add jar
[19:53:31] <nuria__>	 ok, let's update the docs ( i will do it)
[19:58:54] <ottomata>	 k
[20:00:34] <nuria__>	 ottomata, Ironholds : docs for udfs & hive updated: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/QueryUsingUDF
[20:01:05] <ottomata>	 nuria__: you can't select additional columns?
[20:01:08] <ottomata>	 Ironholds: ^
[20:01:09] <Ironholds>	 and the output is?
[20:01:11] <Ironholds>	 yeah, I saw
[20:01:15] <ottomata>	 i got a query to put in there...
[20:01:44] <nuria__>	 ottomata: when using a udf  last time i tested you couldn't
[20:02:00] <nuria__>	 ottomata: I can test again
[20:08:21] <nuria__>	 Ironholds: added sample output
[20:08:42] <Ironholds>	 ta
[20:09:03] <Ironholds>	 oh. balls.
[20:09:10] <Ironholds>	 well, not balls. hmn.
[20:10:09] <ottomata>	 Ironholds: https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/QueryUsingUDF#Example:_User_Agent_UDF
[20:10:13] <ottomata>	 me too!
[20:10:24] <ottomata>	 maybe you can't use them with other columns?  but you can use them in subselects!
[20:10:36] <Ironholds>	 in an ideal world it'd by possible to create new columns from each output, but this isn't an ideal world. This is still awesome :D
[20:10:50] <Ironholds>	 I'm going to build a function for splitting those into a table at the R end right now
[20:11:28] <ottomata>	 you can access them Ironholds
[20:11:37] <ottomata>	 ua(user_agent)['device_family']
[20:11:49] <Ironholds>	 aha
[20:12:21] <Ironholds>	 but then if I go ua(user_agent)['device_family'] AS device, ua(user_agent)['os_family'] AS OS
[20:12:24] <Ironholds>	 will that run once or twice?
[20:12:42] <Ironholds>	 IOW, am I limited to ua(user_agent)['device_family','os_family']?
[20:12:53] <ottomata>	 i'm not sure, i haven't played with this much, see what you can do.
[20:13:03] <Ironholds>	 well, it's hard for me to performance test at this end ;p
[20:13:12] <Ironholds>	 I guess I can just run each in turn and see what the runtime difference is
[20:13:14] <ottomata>	 you could do
[20:14:19] <Ironholds>	 okay, lesse what happens...
[20:15:06] <ottomata>	 maybe htis?
[20:15:06] <ottomata>	 select a.parsed_ua['os_family'], a.parsed_ua['device_family'] from (
[20:15:06] <ottomata>	     select ua(user_agent) as parsed_ua from webrequest
[20:15:06] <ottomata>	     where year=2014 and month=10 and day=30 and hour=0 and webrequest_source='mobile' limit 10) a
[20:15:06] <ottomata>	 ;
[20:15:07] <ottomata>	 trying it
[20:15:32] <nuria__>	 Ironholds: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-GROUPingandSORTingonf(column)
[20:15:34] <ottomata>	 btw, other columns work fine with this udf
[20:15:36] <ottomata>	 nuria__:
[20:15:46] <ottomata>	  select uri_path, ua(user_agent)['os_family'] was fine
[20:15:57] <ottomata>	 yeah, oliver that works
[20:15:58] <ottomata>	 ^^^
[20:15:58] <nuria__>	 ottomata: right, i just run query too
[20:16:37] <ottomata>	 oh you can do it without subquery if you don't use alias
[20:16:39] <ottomata>	 huh
[20:16:43] <ottomata>	 i guess hive jsut figures out what you mean
[20:16:43] <ottomata>	 hm
[20:17:37] <Ironholds>	 ottomata, yeah, I'm not seeing any real performance difference
[20:17:37] <Ironholds>	 cool!
[20:17:59] <nuria__>	 ottomata: docs corrected
[20:18:30] <Ironholds>	 I'll build the parser anyway. It's nice to have an excuse to code while waiting for queries to run ;p
[20:19:00] <ottomata>	 oh, nuria__there are two seconds of that doc that uaparserudf
[20:19:03] <ottomata>	 sections*
[20:19:09] <ottomata>	 https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/QueryUsingUDF#Sampling_Data:_Get_a_user_agent_report_for_the_past_month
[20:19:41] <ottomata>	 we should probably merge this page into the Hive/Queries page
[20:20:25] <nuria__>	 ottomata: i linked it from there a while back right?
[20:20:31] <nuria__>	 leemme see
[20:20:42] <nuria__>	 ottomata: it is linked
[20:21:09] <ottomata>	 mk, i think we should merge them, but i'm not gonna do it (at least not now!) :)
[20:22:19] <nuria__>	 Ottomata: i leave it up to you but I think is fine, we can use this one to document the udfs available
[20:23:19] <Ironholds>	 would there be any way for those to just be auto-loaded when hive is?
[20:23:27] <Ironholds>	 unless there's a measurable cost to more java
[20:24:26] <ottomata>	 one day Ironholds
[20:24:28] <ottomata>	 need new version of hive
[20:24:49] <Ironholds>	 ottomata, so clearly you should upgrade! you love upgrading everything so very much! :D
[20:25:08] <ottomata>	 ha, indeed!
[20:25:11] <ottomata>	 it will happen
[20:25:18] <ottomata>	 first we have to upgrade the whole cluster to trusty
[20:25:25] <ottomata>	 then we upgrade the whoel cluster to cdh 5.2
[20:25:27] <ottomata>	 it will happen!
[20:25:35] <ottomata>	 but i want to focus on some other things for now
[20:25:57] <nuria__>	 also doesn't seem such a good idea to load a bunch of stuff you might not need
[20:26:10] <nuria__>	 i do not think naming functions is actually bad
[20:31:20] <Ironholds>	 well, so far it's just two functions
[20:31:31] <Ironholds>	 and in exchange for doing that, researchers have to remember "ua" and "geolocate"
[20:32:02] <grrrit-wm>	 (PS2) Milimetric: Transform projectcounts hourly files [analytics/refinery] - https://gerrit.wikimedia.org/r/169974
[20:32:04] <Ironholds>	 instead of the URL, or CREATE TEMPORARY FUNCTION ua as 'org.wikimedia.analytics.refinery.hive.UAParserUDF'; and the geo equivalent
[20:36:48] <nuria__>	 Ironholds:  is the geo udf merged already? i think i still WIP
[20:37:08] <Ironholds>	 nuria__, to my knowledge you're right, I just load it from otto's testbed area ;p
[20:37:40] <nuria__>	 Ironholds: which means that even if upgrades you will still need to add it by hand
[20:37:58] <nuria__>	 as it is not available in archiva yet
[20:38:11] <nuria__>	 Ironholds: makes sense?
[20:38:15] <Ironholds>	 yes, I was being sarcastic about the upgrades, referencing ottomata's stress of upping to CDH5 and then upgrading stat100* to Trusty
[20:38:23] <nuria__>	 ahhh
[23:20:14] <bmansurov>	 Hello nice wikimetrics devs. I'm having a problem getting vagrant working when wikimetrics is enabled. Can anybody help? This is the error message I'm getting: Error: Invalid parameter db_user_centralauth at /tmp/vagrant-puppet-5/modules-0/role/manifests/wikimetrics.pp:70 on node mediawiki-vagrant.dev
[23:36:49] <nuria__>	 bmansurov: you need to have the centralauth role enabled  + upgrade vagrant/puppet
[23:37:07] <nuria__>	 bmansurov: vagrant/puppet/wikimetrics
[23:37:20] <bmansurov>	 nuria__: thanks
[23:37:40] <nuria__>	 puppet will do it for you
[23:37:48] <nuria__>	 if you run vagrant provision
[23:37:54] <nuria__>	 bmansurov^
[23:38:02] <nuria__>	 so no need to do it buy hand
[23:38:06] <bmansurov>	 nuria__: cool
[23:38:06] <nuria__>	 *by hand
[23:38:17] <nuria__>	 bmansurov: but you need to have latest puppet vagrant
[23:38:36] <nuria__>	 bmansurov: ok, lemme know if you run into issues
[23:39:06] <bmansurov>	 nuria__: ok, i'm trying now
[23:39:22] <nuria__>	 bmansurov: do you use vagrant for mediawiki?
[23:39:30] <bmansurov>	 nuria__: yes
[23:39:37] <nuria__>	 bmansurov: the way it works is very similar
[23:39:48] <nuria__>	 bmansurov: ok, then you know how to do all this
[23:45:22] <bmansurov>	 nuria__: So I enabled 'centralauth', and ran 'vagrant provision', this should upgrade puppet, right?
[23:45:43] <nuria__>	 bmansurov: actually puppet run enables "centraauth"
[23:45:49] <nuria__>	 so need to do  it by hand
[23:45:53] <nuria__>	 but it works too
[23:46:00] <nuria__>	 but no
[23:46:11] <nuria__>	 bmansurov: vagrant provision does not upgrade puppet
[23:46:20] <nuria__>	 bmansurov: you need to git pull in the depo
[23:46:27] <nuria__>	 bmansurov: making sure you are in master
[23:46:32] <nuria__>	 also
[23:46:39] <bmansurov>	 you mean the puppet repo?
[23:47:02] <nuria__>	 yes, /vagrant one
[23:47:26] <nuria__>	 the vagrant repo has a puppet directory inside
[23:47:41] <bmansurov>	 nuria__:  so I'm in the /vagrant/puppet folder and pulled master
[23:47:51] <bmansurov>	 nuria__: what command do I run now?
[23:48:17] <nuria__>	 git submodule update --init , just like you would do for mediawiki
[23:48:23] <nuria__>	 and after vagrant provision
[23:48:31] <nuria__>	 bmansurov:
[23:49:21] <bmansurov>	 nuria__: thanks, I think it worked