[13:50:05] <ottomata>	 moorning!
[13:50:15] <ottomata>	 drdee, is that varnishkafka check in supposed to be daily?
[13:50:22] <drdee>	 moorning!
[13:50:47] <drdee>	 yup
[13:51:28] <drdee>	 has kripke been salvaged?
[13:53:29] <ottomata>	 yup!  i think dan has some work to do making the limn instances on limn0 work properly
[13:53:34] <ottomata>	 i think right now you can't save graphs or something
[13:53:41] <ottomata>	 but
[13:53:41] <ottomata>	 the ip has been moved
[13:53:57] <ottomata>	 so nexus. and all the limn instances are hosted on limn0 now
[13:54:01] <ottomata>	 i'd like to move nexus elsewhere
[13:54:05] <ottomata>	 puppetized, etc.
[13:54:10] <ottomata>	 but for now I just copied it
[13:57:23] <drdee>	 yup, can't we reuse someone else's puppet manifest for this?
[13:57:28] <ottomata>	 probably
[13:57:30] <ottomata>	 haven't looked
[13:57:34] <ottomata>	 its real simple though
[13:57:38] <ottomata>	 but
[13:57:40] <ottomata>	 dunno about a .deb
[13:57:45] <ottomata>	 that would make the puppet stuff more difficult
[13:57:46] <ottomata>	 as usual
[13:58:01] <ottomata>	 i'd be fine with not puppetizing the installation
[13:58:13] <ottomata>	 we could host this on stat1001, no probs
[13:58:38] <ottomata>	 i'm going to take hadoop down for a sec
[13:58:42] <ottomata>	 need to restart an10
[13:58:45] <drdee>	 well nexus does need a of diskspace
[13:58:51] <drdee>	 need a lot
[13:58:53] <drdee>	 i meant
[13:58:54] <drdee>	 ok
[13:59:03] <ottomata>	 4.1T avail on stat1
[13:59:10] <ottomata>	 stat1001
[13:59:25] <ottomata>	 also, stat1002 is ready again, with new disks
[13:59:27] <ottomata>	 i can work on that too
[13:59:43] <ottomata>	 going to finish this kernel upgrade stuff, i'll need to reboot udp2log hosts too
[13:59:58] <ottomata>	 i might wait for oxygen until we do the precise upgrade
[14:00:07] <ottomata>	 then i'm going to work on hadoop cdh4
[14:00:11] <ottomata>	 then maybe stat1002
[14:00:13] <ottomata>	 then kafka
[14:00:15] <ottomata>	 s'ok?
[14:00:56] <ottomata>	 !log stopping hadoop and restarting analytics1010 (namenode) for kernel upgrade
[14:00:58] <analytics-logbot>	 Logged the message, Master
[14:09:11] <drdee>	 sounds great!
[14:28:40] <gerrit-wm>	 New review: Diederik; "Ok." [analytics/wikistats] (master) C: 2;  - https://gerrit.wikimedia.org/r/62209
[14:28:40] <gerrit-wm>	 Change merged: Diederik; [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/62209
[14:29:03] <gerrit-wm>	 New review: Diederik; "Ok." [analytics/wikistats] (master) C: 2;  - https://gerrit.wikimedia.org/r/62206
[14:29:04] <gerrit-wm>	 Change merged: Diederik; [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/62206
[15:00:05] <ottomata>	 drdee, whachu think
[15:00:29] <ottomata>	 can I reboot gadolinium?  the multicast stream would be offline for maybe 5 mins or less
[15:54:18] <ottomata>	 !log rebooted gadolinium to upgrade linux kernel
[15:54:20] <analytics-logbot>	 Logged the message, Master
[16:01:48] <jeremyb>	 ottomata: got your mail in duplicate fyi.
[16:02:47] <ottomata>	 yeah, i accidentally sent the first with my personal gmail
[16:02:51] <ottomata>	 resent from wmf addy
[16:07:48] <jeremyb>	 ahhh
[16:07:52] <jeremyb>	 well i got both
[16:37:02] <drdee>	 ottomata, can we do scrum now?
[16:37:16] <drdee>	 erosen: scrum now?
[16:37:22] <drdee>	 it's just the three of us
[16:37:22] <erosen>	 sure
[16:37:24] <erosen>	 works for me
[16:38:13] <ottomata>	 sure
[16:45:35] <ottomata>	 drdee
[16:45:36] <ottomata>	 !
[16:45:38] <ottomata>	 we are waitinf o youuu
[16:46:49] <drdee>	 sorrryyyyyy
[17:32:20] <erosen>	 ottomata: re 244, looks like what I suspected is happening: the country counts are only being generated from requests which are tagged with x-cs
[17:32:23] <erosen>	 ottomata: see https://github.com/wikimedia/kraken/blob/master/pig/webrequest_zero_hour_carrier_country.pig#L34
[17:32:28] <erosen>	 i think the fix is simple
[17:32:34] <erosen>	 but i will mean we need to rerun thing
[17:32:34] <erosen>	 s
[17:32:53] <ottomata>	 oh
[17:32:54] <ottomata>	 yes
[17:32:57] <ottomata>	 that's correct
[17:33:06] <ottomata>	 was something different supposed to happen?
[17:33:20] <erosen>	 well the country counts are meant to be the total number of requests from that coutry
[17:33:22] <ottomata>	 i thought we weren't going to trust zero stats in kraken until we sorted out the x-analytics problems
[17:33:26] <ottomata>	 oh
[17:33:30] <erosen>	 so that we can assess the relative share which a particular provider has
[17:33:51] <ottomata>	 huh, ok so different jobs
[17:33:54] <erosen>	 well, regarding the x-analtyics problems, we think they don't exist...
[17:34:01] <ottomata>	 right ok
[17:34:05] <erosen>	 well they don't need to be different jobs
[17:34:17] <erosen>	 i think we can just reorder the pig script
[17:34:22] <ottomata>	 hm
[17:34:25] <erosen>	 and generate the country counts prior to the filter command
[17:34:27] <erosen>	 maybe ...?
[17:35:02] <ottomata>	 might make the job take longer
[17:35:05] <ottomata>	 but should work
[17:35:10] <ottomata>	 also, i just noticed this:
[17:35:11] <erosen>	 i'm happy to split the job up
[17:35:13] <ottomata>	 carrier_count = FOREACH (GROUP log_fields BY (date_bucket, language, project, site_version, country, carrier))
[17:35:14] <ottomata>	 is that ok?
[17:35:22] <ottomata>	 country,carrier?
[17:35:29] <ottomata>	 that means that if there are different countries for a single carrier
[17:35:35] <ottomata>	 they will be counted differently in the carrier_count
[17:35:37] <ottomata>	 is that ok?
[17:35:40] <erosen>	 yeah
[17:35:42] <ottomata>	 ok
[17:35:53] <erosen>	 I do a groupby on the data before I use it
[17:35:58] <ottomata>	 ok
[17:35:59] <erosen>	 so it is always okay to have an extra field in there
[17:36:10] <ottomata>	 hm
[17:36:32] <erosen>	 what vexes you?
[17:36:46] <ottomata>	 naw, got it, was going to ask another q but i got it
[17:36:48] <ottomata>	 ok so hm
[17:37:09] <ottomata>	 i wonder if we just change the pig script, if the job will just start using it
[17:37:29] <erosen>	 seems likely
[17:37:48] <erosen>	 we can manually run it on a smaller date range
[17:37:57] <erosen>	 and then switch out the file which oozie uses
[17:39:17] <ottomata>	 eh?
[17:39:34] <ottomata>	 gonna try this...
[17:40:25] <ottomata>	 hmmm actually
[17:40:30] <ottomata>	 i think this should be a diffferent job
[17:40:36] <ottomata>	 this is full mobile country counts then
[17:40:38] <ottomata>	 right?
[17:40:46] <ottomata>	 not necessarily anything specific to zero
[17:41:08] <erosen>	 yup
[17:41:08] <erosen>	 exactly
[17:41:38] <ottomata>	 do you still want this bit of the filter?
[17:41:38] <ottomata>	      - host matches *.wikipedia.org
[17:41:38] <ottomata>	      - request is a pageview
[17:41:50] <ottomata>	 for country count?
[17:41:53] <erosen>	 hmm
[17:42:06] <erosen>	 from a greedy "more info is better" perspective
[17:42:13] <erosen>	 i think we should keep the project in the output
[17:42:25] <ottomata>	 yeah i think so to
[17:42:26] <ottomata>	 but
[17:42:27] <erosen>	 by project i mean {'wikipedia', 'meta.wikimedia', 'wikisource'}
[17:42:34] <ottomata>	 right now those are being filtered out
[17:42:38] <erosen>	 but it is fine for now if it is easier
[17:42:38] <gerrit-wm>	 New patchset: Erik Zachte; "Collect and use countable namespaces via api" [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/64598
[17:42:42] <erosen>	 yeah
[17:42:45] <ottomata>	      - host matches *.wikipedia.org
[17:42:54] <ottomata>	 naw, its the same if I make it a different job
[17:43:00] <ottomata>	 what about pageview?
[17:43:04] <ottomata>	      - request is a pageview
[17:43:11] <erosen>	 yeah
[17:43:14] <erosen>	 that part should stay
[17:43:30] <erosen>	 we are trying to use that definition more broadly
[17:44:11] <gerrit-wm>	 Change merged: Erik Zachte; [analytics/wikistats] (master) - https://gerrit.wikimedia.org/r/64598
[17:44:33] <ottomata>	 k
[17:44:38] <ottomata>	 so
[17:44:44] <ottomata>	 filter for pageviews
[17:44:48] <ottomata>	 group by same fields
[17:44:49] <ottomata>	 output
[17:45:01] <erosen>	 yup
[17:45:01] <ottomata>	 hm
[17:45:02] <ottomata>	 ok
[17:45:08] <ottomata>	 i think i will keep this at the same location, as a zero job
[17:45:11] <ottomata>	 but I will make it another job
[17:45:12] <erosen>	 check this out: https://mingle.corp.wikimedia.org/projects/analytics/cards/244
[17:45:21] <erosen>	 the output format is pretty well specified
[17:45:24] <erosen>	 in the example
[17:47:49] <erosen>	 ottomata: are you working on this?
[17:47:58] <erosen>	 ottomata: in general, what is your pig dev env like?
[17:48:04] <ottomata>	 yeah, working on it
[17:48:18] <ottomata>	 in general: i don't have one, pig testing I do from an02
[17:48:20] <ottomata>	 in grunt usually
[17:48:28] <ottomata>	 we hope to have one in labs eventually
[17:48:28] <erosen>	 gotcha
[17:48:42] <erosen>	 well then it would need the data, right?
[17:48:48] <ottomata>	 i do have a vagrant vm that I will keep tweaking too, i'm using it for puppet dev, but could also use it for general kraken dev
[17:48:58] <ottomata>	 it would need fake data
[17:48:58] <erosen>	 that's roughly what I would do--just checking
[17:48:59] <erosen>	 ya
[17:52:20] <erosen>	 ottomata: there is another issue which we may want to fix at the same time: the x-cs code -> name lookup is still a bit rough
[17:52:37] <erosen>	 and the tata-india stuff is missing for some reason
[17:53:10] <ottomata>	 ungh, that's the udf?
[17:53:19] <erosen>	 not sure
[17:53:31] <erosen>	 i've been trying to find it for the last few minutes
[17:53:49] <erosen>	 i think when we designed the system we created a json file which represents the mapping
[17:53:58] <erosen>	 and that then gets loaded in some UDF
[17:54:01] <erosen>	 but I don't know which one
[17:54:13] <ottomata>	 Zero.java
[17:54:19] <erosen>	 yeah, where is that?
[17:54:27] <erosen>	 aah, not in pig
[17:54:27] <ottomata>	 kraken-pig/...
[17:54:59] <erosen>	 https://github.com/wikimedia/kraken/blob/master/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java
[17:55:21] <ottomata>	 ja
[17:56:50] <erosen>	 hmm
[17:57:03] <erosen>	 do you know where that UDF expects the actual file to live?
[17:57:19] <ottomata>	 ?
[17:57:35] <ottomata>	 this?
[17:57:35] <ottomata>	         mccMncMap = converter.construct("org.wikimedia.analytics.kraken.schemas.MccMnc", "mcc_mnc.json", "getMCC_MNC");
[17:57:36] <erosen>	 https://github.com/wikimedia/kraken/blob/master/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java#L61
[17:57:38] <erosen>	 hehe
[17:57:38] <erosen>	 yeah
[17:57:49] <ottomata>	 drdee maybe does
[17:57:51] <jgonera>	 hey, does someone know where I can find the source code of http://test-reportcard.wmflabs.org/graphs/active_editors_target (I don't mean Limn, but the datasources and graphs)?
[17:57:51] <erosen>	 what is the pwd in that case?
[17:57:52] <ottomata>	 i've never looked at this
[17:58:27] <erosen>	 jgonera: you should be able to access them at  http://test-reportcard.wmflabs.org/graphs/active_editors_target.json
[17:59:12] <erosen>	 and http://test-reportcard.wmflabs.org/datasources/rc_active_editors_target_aligned.json
[17:59:24] <ottomata>	 erosen, i think its in kraken.jar
[17:59:41] <erosen>	 jgonera: add ?pretty=True to have it pretty printed like this: http://test-reportcard.wmflabs.org/datasources/rc_active_editors_target_aligned.json?pretty=True
[17:59:47] <erosen>	 ottomata: intresting
[18:01:04] <jgonera>	 erosen, thanks, I think I've found what I needed
[18:01:09] <erosen>	 cool
[18:11:42] <jgonera>	 erosen, do you know if I can make something like this: instead of bars from consecutive columns in the data table being drawn on top of one another in z-axis, draw them on top of one another in y-axis?
[18:11:48] <jgonera>	 this probably sounds confusing
[18:12:20] <erosen>	 hehe
[18:12:22] <erosen>	 a litt
[18:12:23] <erosen>	 le
[18:12:32] <erosen>	 i think i know what you mean though
[18:12:36] <erosen>	 and I think the answer is yes
[18:12:50] <erosen>	 does the idea of a "stacked" chart sort of fit what you are thinking?
[18:12:57] <jgonera>	 hm, let's say we have a table with three columns: date, apples, oranges, with the following rows: (1/1/2013, 5, 10), (2/1/2013, 7, 4)
[18:13:01] <jgonera>	 I guess
[18:13:05] <erosen>	 so that the plotted value of a particular line is the cumulative value of all the lines beneath it?
[18:13:29] <jgonera>	 yeah, I'd just prefer bars to lines though ;)
[18:15:04] <erosen>	 yeah that is possible
[18:15:09] <jgonera>	 so in the apples and oranges example, the oranges bar would be stacked on top of the apples bar
[18:15:09] <erosen>	 i believe
[18:15:27] <erosen>	 finding an example
[18:15:34] <ottomata>	 lunchtime!  back in a bit
[18:15:34] <jgonera>	 thanks
[18:15:51] <erosen>	 jgonera: like this: http://gp.wmflabs.org/graphs/grants_count_by_global_south?
[18:16:51] <jgonera>	 erosen, not really, there if "Number of Grants by Global South, north" is set to 13 it will completely cover "Number of Grants by Global South, South"
[18:17:04] <erosen>	 really?
[18:17:08] <erosen>	 interesting...
[18:17:18] <jgonera>	 in other words, this is based on the fact that we can predict which column has the higher value
[18:17:33] <erosen>	 i see
[18:17:36] <erosen>	 very true
[18:17:45] <erosen>	 well the order actually doesn't matter in the datasource
[18:17:52] <erosen>	 but i believe it is possible to make it stacked
[18:18:35] <jgonera>	 it matters in the graph JSON file, I checked it, Limn doesn't automatically determine which column has the lowest value
[18:18:41] <jgonera>	 and in my case, this can change over time anyway
[18:18:43] <erosen>	 this might be more useful http://reportcard.wmflabs.org/graphs/pageviews_mobile_target_stacked
[18:19:58] <jgonera>	 hah, this might be it! ;) "stack": {
[18:19:58] <jgonera>	                         "enabled": true
[18:19:58] <jgonera>	                     }
[18:20:08] <jgonera>	 thanks, I'll try to play with it
[18:20:15] <erosen>	 yeah i think that's the trick
[18:22:40] <jgonera>	 yep, that works, for some reason though the y-axis range is not auto-adjusted now, but I'll try to figure this out
[18:22:53] <erosen>	 yeah, i think that is a known bug
[18:23:15] <jgonera>	 do you know if I can somehow show the cumulative value in the legend when I hover the bars?
[18:23:17] <jgonera>	 oh
[18:23:24] <erosen>	 not sure
[18:24:02] <erosen>	 really milimetric is the one to ask
[18:24:21] <erosen>	 he isn't around today, but he should be around later this week
[18:24:46] <jgonera>	 ok,I'll ask him when I see him, the stacked bars already solve a lot for me, thanks ;)
[18:24:51] <erosen>	 great
[18:57:53] <ottomata>	 baack
[19:07:50] <ottomata>	 erosen, do you need/want historical for this new job?
[19:07:53] <ottomata>	 zero_country?
[19:08:11] <ottomata>	 the existing zero/country/ data is inaccurate, right?
[19:09:35] <erosen>	 ottomata: yes, existing data is inaccurate
[19:09:44] <erosen>	 i will eventually need new data
[19:09:47] <ottomata>	 k
[19:09:49] <erosen>	 but let's just do a sample
[19:09:52] <erosen>	 like the last month
[19:09:54] <erosen>	 is that easy?
[19:10:06] <ottomata>	 yeah, think so
[19:10:30] <ottomata>	 you mean april or may?
[19:15:02] <erosen>	 how about april
[19:15:11] <erosen>	 ottomata: sorry, didn't see this ^^ till now
[19:15:44] <ottomata>	 ok
[19:15:45] <ottomata>	 s'ok
[19:15:53] <ottomata>	 this is going to take a bit of playing and tweaking
[19:15:58] <ottomata>	 oozie is slow going for dev :)
[19:16:07] <ottomata>	 i'm not sure what you want done to Zero.java though
[19:16:48] <erosen>	 i'm building a new mcc-mnc.json file
[19:16:55] <erosen>	 and then we can use that one
[19:17:06] <erosen>	 we can work on the two issues separately
[19:31:04] <erosen>	 ottomata: is it simple to disable the translation of x_cs to carrier name?
[19:32:03] <ottomata>	 ?
[19:32:06] <ottomata>	 i know nothing!
[19:32:08] <erosen>	 hehe
[19:32:17] <erosen>	 this line: FLATTEN(ZERO(x_cs))         AS (carrier:chararray, carrier_iso:chararray)
[19:32:26] <ottomata>	 you want to remove that?
[19:32:27] <ottomata>	 eyah totally
[19:32:29] <erosen>	 ideally we could just extract the x_cs line
[19:32:31] <ottomata>	 do you want to just group by x-cs
[19:32:33] <ottomata>	 yeah no probs
[19:32:43] <erosen>	 i meant ideally we could extract the x_cs **code**
[19:32:55] <erosen>	 which I think the zero udf does
[19:33:03] <erosen>	 i'll find a better line to send you
[19:35:59] <ottomata>	 hm ok
[19:36:00] <ottomata>	 oh
[19:36:12] <ottomata>	 k
[19:36:15] <erosen>	 okay, nvm it is too much work i think
[19:37:23] <erosen>	 we could replace this line: https://github.com/wikimedia/kraken/blob/master/kraken-pig/src/main/java/org/wikimedia/analytics/kraken/pig/Zero.java#L91 with this: getMCC_MNC()
[19:37:28] <erosen>	 but i don't really like that solution
[19:37:48] <erosen>	 i just can't decide whether the number -> name lookup should happen in kraken or python