[01:46:38] <gerrit-wm>	 New patchset: Rfaulk; "mod. (bz 48341) remove 'ban' field add 'is_blocked' (bool)." [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/70117
[01:46:58] <gerrit-wm>	 Change merged: Rfaulk; [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/70117
[09:19:54] <gerrit-wm>	 New patchset: Rfaulk; "mod. (bz 48341) modify datatype indices and rate aggregator value index." [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/70129
[09:20:13] <gerrit-wm>	 Change merged: Rfaulk; [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/70129
[11:53:47] <milimetric>	 morning
[12:44:44] <gerrit-wm>	 New patchset: Stefan.petrea; "test" [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/70153
[12:46:09] <gerrit-wm>	 New patchset: Stefan.petrea; "Changed start of processing for new mobile" [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/70153
[13:00:06] <gerrit-wm>	 Change merged: Stefan.petrea; [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/70153
[13:29:48] <gerrit-wm>	 New patchset: Stefan.petrea; "Removed obsolete links" [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/70158
[13:31:17] <gerrit-wm>	 Change merged: Stefan.petrea; [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/70158
[14:57:10] <drdee>	 hihaaaaaaaa
[14:57:17] <drdee>	 ottomata in tha house?
[14:57:25] <ottomata>	 yoyooyoooo
[14:57:25] <drdee>	 milimetric in thaaa house???
[14:57:31] <ottomata>	 I AM IN THAA AHOUUUUSE
[14:57:36] <drdee>	 ALLLRIGHT
[14:57:40] <drdee>	 LET"S KICK IT
[14:57:53] <milimetric>	 aite, kick it dr dee
[14:58:31] <drdee>	 yo we are one team,
[14:58:34] <drdee>	 focused like a beam
[14:58:51] <drdee>	 counting through the days
[14:58:54] <drdee>	 in many ways
[15:03:18] <drdee>	 milimetric what's the name of the balsamic component that you showed me on  friday that does the tab thing for metric parameters?
[15:03:43] <milimetric>	 oh nice rhyme above
[15:03:51] <milimetric>	 cool, this day's getting off to a good start
[15:03:59] <milimetric>	 it's just in the containers tab
[15:04:11] <milimetric>	 vertical tabs or something like that?
[15:05:14] <drdee>	 k
[15:05:20] <drdee>	 got it
[15:05:42] <milimetric>	 cool, not sure if we'll do it exactly that way, but it'll be good for the mockup
[15:11:11] <drdee>	 i am working on the mockup right now
[15:11:27] <drdee>	 i have a solution for the time series data btw
[15:14:36] <drdee>	 milimetric: we should make a doc with parameters per metric, do we already have such a doc?
[15:15:09] <milimetric>	 nope
[15:15:15] <milimetric>	 but i would disagree
[15:15:27] <milimetric>	 we should iterate on this
[15:15:39] <milimetric>	 start out with something that seems logical, and add parameters as we find them
[15:15:58] <drdee>	 sure we should iterate but we can already take the params from umapi v1, right?
[15:15:58] <milimetric>	 because it seems to me that the parameters were designed before the requirements
[15:16:09] <milimetric>	 that's what i'm not sure about
[15:16:20] <milimetric>	 whether or not those parameters were meeting a requirement
[15:16:37] <milimetric>	 but if Dario needs them, he'll just say - migrate as is
[15:16:38] <milimetric>	 and we'll do that
[15:16:53] <milimetric>	 for the first two metrics, it's pretty easy
[15:17:08] <milimetric>	 (Just namespace edits and bytes added)
[15:33:57] <drdee>	 milimetric, batcave?
[15:34:03] <milimetric>	 omw
[15:38:16] <ottomata>	 drdee hiya, any word on the zero country thing?  you just needed to upload the new kraken jars, right?
[15:38:28] <drdee>	 can do that in a sec
[16:00:31] <drdee>	 ottomata: https://plus.google.com/hangouts/_/2763caa5549d08d05417702af2d1bfa572601d0a
[16:00:34] <drdee>	 varnishkafka
[16:00:42] <ottomata>	 oop
[16:00:42] <ottomata>	 cpoming
[16:08:18] <gerrit-wm>	 New patchset: Milimetric; "fixed re-routing after login and string concat problem" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70193
[16:08:38] <gerrit-wm>	 Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70193
[16:55:09] <drdee>	 milimetric, what was maryana's card regarding commons stuff?
[16:55:43] <milimetric>	 775 off the top of my head
[16:55:59] <milimetric>	 nope
[16:56:35] <milimetric>	 774 drdee
[16:56:36] <milimetric>	 so close
[16:56:42] <drdee>	 thx
[16:58:42] <drdee>	 are you sure? that's about cross wiki edits
[17:12:16] <gerrit-wm>	 New patchset: Milimetric; "fixed metrics test to fail" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70211
[17:12:31] <gerrit-wm>	 Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70211
[17:31:20] <drdee>	 rounce123, tnegrin: https://plus.google.com/hangouts/_/b4a3abe3625563b41964c8f613c20522c49aceb1
[18:15:24] <gerrit-wm>	 New patchset: Erosen; "fixes tests which require logging in" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70220
[18:19:37] <gerrit-wm>	 Change merged: Erosen; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70220
[18:21:53] <ottomata>	 drdee, i'm not sure why (looking into it)
[18:21:54] <ottomata>	 but i'm getting
[18:21:57] <ottomata>	 Error Message     : UndeclaredThrowableException: null
[18:22:02] <ottomata>	 when I try to run zero country
[18:22:34] <ottomata>	 no logs :/
[18:22:48] <drdee>	 grumble
[18:31:15] <ottomata>	 drdee
[18:31:16] <ottomata>	 2013-06-24 18:31:08,550 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Expected (chararray), input does not have 7 fields
[18:31:16] <ottomata>	 Details at logfile: /home/otto/scr/pig/pig_1372098664792.log
[18:31:41] <drdee>	 ohhhhh you are giving the wrong input
[18:34:00] <drdee>	 i thought i had pushed an updated pig script
[18:37:11] <ottomata>	 i've got the latest
[18:44:04] <drdee>	 which machine is the log file?
[18:44:26] <drdee>	 and which pig script did you use? (where can i find it)?
[18:49:55] <milimetric>	 hey rounce123, I see tnegrin isn't in IRC
[18:50:24] <milimetric>	 if he's having trouble with that, let him know how you set it up
[18:50:35] <rounce123>	 Think he is in meetings now
[18:50:41] <rounce123>	 will check with him
[18:52:06] <drdee>	 ottomata: ^^
[18:54:17] <ottomata>	 drdee, oozie just failed without giving the error, so I ran like this:
[18:55:22] <drdee>	 ....
[18:55:49] <ottomata>	 pig -f /opt/kraken/pig/zero_country.pig -p input=/wmf/raw/webrequest/webrequest-wikipedia-mobile/dt=2013-06-01_* -p output=tmp/zero_country_outA -p date_regex_format=2013-06-01
[18:55:53] <ottomata>	 on an02
[18:57:57] <drdee>	 where are you getting the jars from?
[18:58:11] <drdee>	 it seems like an old jar
[18:58:49] <drdee>	 ottomata: ^^
[18:59:36] <ottomata>	 HMMMMmmMoops yeah sorry, i had just run it in my homedir thikning it would pull out of /libs...
[18:59:36] <ottomata>	 sorry
[18:59:39] <ottomata>	 thanks
[18:59:40] <ottomata>	 lemme try again
[19:03:59] <ottomata>	 yargh, is this my fault too drdee?
[19:04:07] <ottomata>	 2013-06-24 19:02:16,514 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: could not instantiate 'org.wikimedia.analytics.kraken.pig.GeoIpLookupEvalFunc' with arguments '[countryCode, GeoIPCity]'
[19:04:07] <drdee>	 what?
[19:04:17] <ottomata>	 i've got the geoip.jar
[19:04:18] <ottomata>	 gr
[19:04:21] <ottomata>	 geoip-1.2.9-patch-2-SNAPSHOT.jar
[19:04:23] <ottomata>	 right?
[19:04:24] <drdee>	 yes, that used to work all the time
[19:04:45] <ottomata>	 ok still looking
[19:09:36] <drdee>	 milimetric:  pull from wikimetrics ;)
[19:09:51] <milimetric>	 k
[19:10:23] <milimetric>	 very cool, but i'll need my windows compy to open them i think
[19:11:18] <drdee>	 k
[19:14:35] <ottomata>	 erosen, sorry, i'm still trying to get zero country running, hopefully will have time to brainbounce soon, i'm going to take a 15 min power nap and get some coffee, brb
[19:49:02] <ottomata>	 ohhh drdee!!!
[19:49:09] <drdee>	 yoooo
[19:49:15] <ottomata>	 i just noticed you took out the date_bucket_regex usage out of zero_carrier.pig
[19:49:27] <drdee>	 did I?
[19:49:29] <ottomata>	 yeah
[19:49:43] <ottomata>	 at least
[19:49:44] <ottomata>	 its not there anymore
[19:51:02] <ottomata>	 or maybe it was never there?
[19:51:03] <ottomata>	 looking...
[19:52:23] <drdee>	 it probably was there
[19:54:14] <ottomata>	 yeah
[19:54:18] <ottomata>	 drdee, most recent commit
[19:54:23] <ottomata>	 Wed Jun 19 17:53:40 2013 -0700
[19:54:23] <ottomata>	     Updated Zero Carrier pig script with new business logic.
[19:54:26] <drdee>	 sorrrrryyyyy
[19:54:27] <ottomata>	 -        AND (DATE_BUCKET(timestamp) MATCHES '$date_bucket_regex')
[19:54:39] <ottomata>	 s'ok, gotta re run carrier :):/
[19:56:12] <drdee>	 :(
[19:56:21] <drdee>	 but is the zero_country script now working?
[19:56:53] <ottomata>	 no
[19:56:59] <ottomata>	 dunno what's wrong with that yet
[19:57:02] <ottomata>	 i was comparing the two and just found that
[19:57:11] <drdee>	 on what machine are you doing   your testing?
[19:57:24] <erosen>	 ottomata: drdee: let me know as soon as there is a minimally testable file so that I can let Amit take a look
[19:57:25] <ottomata>	 an02
[19:57:39] <drdee>	 yup, where are the jars that you use?
[19:57:42] <erosen>	 a few days of carrier only would be fine
[19:57:55] <ottomata>	  /home/otto/scr/pig
[19:58:04] <ottomata>	 i got them out of hdfs /libs/kraken-0.0.2
[19:58:17] <drdee>	 the geo stuff should really work
[19:58:19] <ottomata>	 erosen, i'm going to rerun carrier again, should be a few hours to get it caught up to today
[19:58:19] <ottomata>	 yeah
[19:58:24] <ottomata>	 i think so too
[19:58:30] <drdee>	 we haven't touch that in ages
[19:58:39] <ottomata>	 i think i'm doing something wrong there, but that's just because i'm trying to isolate the problem
[19:58:42] <ottomata>	 since oozie isn't tellin gme
[20:00:16] <ottomata>	 also, drdee
[20:00:24] <ottomata>	 carrier is currently using ZERO_PAGEVIEW
[20:00:33] <ottomata>	 DEFINE ZERO_PAGEVIEW  org.wikimedia.analytics.kraken.pig.ZeroFilterFunc('default');
[20:00:36] <ottomata>	 and country is using
[20:00:46] <ottomata>	 IS_PAGEVIEW
[20:00:46] <ottomata>	 DEFINE IS_PAGEVIEW    org.wikimedia.analytics.kraken.pig.PageViewFilterFunc();
[20:00:48] <ottomata>	 is that ok?
[20:01:21] <drdee>	 yes ZERO_PAGEVIEW is a misleading function name
[20:01:45] <drdee>	 it should be something like IS_ZERO_CARRIER
[20:01:55] <drdee>	 you can change that if you wan
[20:01:56] <drdee>	 t
[20:03:20] <erosen>	 milimetric: tell me what you think about this when you get a sec: https://flask-restless.readthedocs.org/en/latest/
[20:03:30] <milimetric>	 yeah, i read it
[20:03:34] <erosen>	 gotcha
[20:03:35] <milimetric>	 not necessary
[20:03:38] <erosen>	 what is your take?
[20:03:45] <erosen>	 k
[20:04:54] <ottomata>	 ok, drdee, that's fine
[20:05:05] <ottomata>	 but, it is ok that the two different metrics are filtering by different thigns?
[20:05:17] <ottomata>	 country just uses IS_PAGEViEW
[20:05:26] <milimetric>	 my guidance has been "go as simple and old-school as possible unless there's a huge benefit"
[20:05:26] <milimetric>	 I liked restless though, it was cool.  Thing is, we aren't actually creating a REST api
[20:05:42] <milimetric>	 we're just using REST convention when creating our api :)
[20:06:01] <erosen>	 hmm
[20:06:29] <erosen>	 just to humor me, why is our API not RESTful?
[20:06:31] <ottomata>	 actually, drdee, as I'm reading zero_country.pig, we're not even filtering for X-analytics header, which is what we used to do
[20:07:29] <drdee>	 org.wikimedia.analytics.kraken.pig.ZeroFilterFunc('default'); is a filter function so we are filtering
[20:07:44] <ottomata>	 that's not being used in zero_country.pig
[20:07:52] <ottomata>	 only in zero_carrier.pig
[20:08:03] <drdee>	 right but we don't need to filter in zero_country
[20:08:11] <ottomata>	 ah this is my question
[20:08:18] <drdee>	 we just need to know the total number of page views in that country to the mobile site
[20:08:53] <ottomata>	 hm, ok, i kinda think that assuming that the input data is all mobile pageviews is a little naive, even though it is true of our data right now
[20:09:05] <ottomata>	 but ok
[20:09:38] <ottomata>	 (didn't mean naive in a mean sense, just that the script won't work when we start importing more data)
[20:10:04] <drdee>	 sure but that will be a new card :D
[20:10:07] <ottomata>	 ok
[20:13:57] <drdee>	 need help with geocoding?
[20:14:57] <ottomata>	 maybe, i'm not sure if that is the problem, see if you can run the pig script with the same parameters I gave you
[20:15:02] <ottomata>	 zero_country.pig
[20:15:07] <ottomata>	 in your own homedir or whatever
[20:18:18] <ottomata>	 ah, HM
[20:18:34] <ottomata>	 drdee, i'm getting the exact same exception from zero_carrier now that I am resubmitting it
[20:18:40] <ottomata>	 so this is not just a zero_country problem
[20:18:42] <ottomata>	 HMMmm
[20:19:05] <drdee>	 well i have been running these scripts for days :)
[20:19:19] <ottomata>	 yeah i think something bigger is wrong...
[20:19:21] <ottomata>	 not sure what yet
[20:20:45] <drdee>	 what is the error log?
[20:21:01] <ottomata>	 Error Message     : UndeclaredThrowableException: null
[20:21:03] <ottomata>	 that's all I ahve
[20:21:21] <ottomata>	 the job doesn't make it to hadoop resourcemanager
[20:21:23] <ottomata>	 pretty sure
[20:21:29] <ottomata>	 i don't have a link,a nd I don't see it in the jobs queue
[20:21:39] <ottomata>	 but oozie accepts it just fine
[20:21:39] <ottomata>	 hm
[20:29:28] <ottomata>	 found more more info, but i'm still confused
[20:30:08] <ottomata>	 Caused by: java.net.ConnectException: Call From analytics1027/10.64.36.127 to analytics1010.eqiad.wmnet:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
[20:30:17] <ottomata>	 hmm
[20:30:30] <ottomata>	 OH
[20:30:32] <ottomata>	 historyserver is down
[20:30:33] <ottomata>	 wha?
[20:32:04] <drdee>	 was history server running on an20?
[20:34:17] <ottomata>	 no
[20:34:18] <ottomata>	 an10
[20:34:51] <ottomata>	 i'll be happy when I get all of the hadoop nodes reinstalled and can do that monitoring card better, eh?
[20:37:39] <drdee>	 aight
[20:46:06] <ottomata>	 yay country works!
[20:46:14] <ottomata>	 doh, sorry that took so long to figure out :/
[20:48:36] <ottomata>	 drdee, still some IPs in the country output, but i guess it doesn't really matter?
[20:48:42] <ottomata>	 http://localhost:8888/filebrowser/view//wmf/public/webrequest/mobile/zero/country/zero_country-daily.tsv
[20:49:00] <drdee>	 mmmmmm it's okay for now
[20:49:03] <drdee>	 we can clean that up late
[20:49:04] <drdee>	 r
[20:59:51] <ottomata>	 k
[21:26:37] <ottomata>	 phew, ok, carrier and country both going now
[21:32:21] <drdee>	 ottomata: awesome!
[21:32:25] <drdee>	 when can erosen check?
[21:32:31] <ottomata>	 in a few hours
[21:32:34] <erosen>	 sounds good
[21:33:59] <drdee>	 milimetric: how do the mockups look?
[21:34:13] <milimetric>	 i haven't looked yet - buried in design
[21:34:23] <milimetric>	 and i figured i'd just be depressed if i looked and saw how much more there is to do :)
[21:35:36] <drdee>	 ottomata: icinga-wm: PROBLEM - Disk space on analytics1018 is CRITICAL: DISK CRITICAL - free space: / 639 MB (2% inode=95%):
[21:40:42] <gerrit-wm>	 New patchset: Milimetric; "job create rough draft at /jobs/create" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70326
[21:40:55] <gerrit-wm>	 Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/70326
[21:41:20] <milimetric>	 drdee: you can pull, and check it out like this:
[21:41:21] <ottomata>	 HMMMM
[21:41:23] <milimetric>	 sudo pip install -e .
[21:41:27] <milimetric>	 wikimetrics web
[21:41:33] <milimetric>	 navigate to /jobs/create
[21:41:40] <ottomata>	 AGHHHH
[21:41:41] <milimetric>	 (click google and accept to login)
[21:41:42] <ottomata>	 oops
[21:41:47] <drdee>	 what's up ottomata
[21:41:53] <milimetric>	 k, gonna run now :)
[21:41:56] <milimetric>	 laters yall
[21:41:59] <drdee>	 laterz
[21:41:59] <ottomata>	 i didn't check on my disk format script well
[21:42:00] <milimetric>	 erosen: see above too
[21:42:03] <ottomata>	 when I brought an18 back up
[21:42:11] <ottomata>	 partitions were created, but not mounted
[21:42:14] <ottomata>	 can fix
[21:42:17] <drdee>	 k
[21:43:59] <erosen>	 milimetric: one sec, figuring out a 2011 tax issue
[22:21:40] <ottomata>	 laters all!