[09:46:58] <grrrit-wm>	 (CR) Nuria: [C: 2] Remove useless scripts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/141076 (owner: Milimetric)
[09:49:08] <grrrit-wm>	 (CR) Nuria: [C: 2] "> This script will allow us to slowly roll it out and we shouldn't need to roll back except if we get in trouble." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/141077 (https://bugzilla.wikimedia.org/65946) (owner: Milimetric)
[09:55:42] <grrrit-wm>	 (CR) Nuria: Fix wiki cohort display (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[10:12:43] <grrrit-wm>	 (CR) Nuria: [C: 2] Ensure wiki cohorts work [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140830 (https://bugzilla.wikimedia.org/66290) (owner: Milimetric)
[10:12:57] <grrrit-wm>	 (Merged) jenkins-bot: Ensure wiki cohorts work [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/140830 (https://bugzilla.wikimedia.org/66290) (owner: Milimetric)
[10:25:09] <grrrit-wm>	 (PS2) Milimetric: Remove redundant outdated secondary log backup [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/141689
[10:25:14] <grrrit-wm>	 (PS2) Milimetric: Add pretty symlink for WikimetricsBot [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087)
[10:25:33] <grrrit-wm>	 (PS2) Milimetric: Fix wiki cohort display [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514
[10:28:51] <grrrit-wm>	 (CR) Milimetric: Fix wiki cohort display (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[10:29:00] <grrrit-wm>	 (PS2) Milimetric: Remove limit on recurrent, add throttling [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142007 (https://bugzilla.wikimedia.org/66841)
[10:29:40] <milimetric>	 hey nuria, sorry I had forgotten to add you to like 3 of the patches I pushed over the last few days
[10:29:52] * milimetric thinks adding people to patches manually is silly
[10:29:57] <nuria>	 np at all
[10:30:19] <nuria>	 i just had not looked at that code for couple of days.
[10:30:32] <milimetric>	 so I'm gonna go bike into the coworking space today and I won't be available for about an hour, but until then I'm around
[10:30:39] <nuria>	 but wait.. is super early in the east coast
[10:30:48] <nuria>	 i hadn't even noticed
[10:30:59] <milimetric>	 i couldn't sleep, it's getting hot here :(
[10:31:01] <milimetric>	 I hate heat
[10:31:53] <milimetric>	 so i guess i'll grab one of the hadoop tasks?
[10:32:21] <milimetric>	 I'll be terrible at it :)  can nobody else like qchris or you work on that instead? :D
[10:32:37] * qchris reads backscroll.
[10:33:28] * qchris is just finishing testing the backup.
[10:33:48] <qchris>	 Afterwards I'll pick up whatever is on the scrum board.
[10:34:13] <qchris>	 milimetric: I do not care much, whether I do Hadoop stuff or other work.
[10:34:26] <qchris>	 Whatever you prefer.
[10:34:40] <milimetric>	 qchris: I saw you made the duplicate user_id bug s=none
[10:34:48] <milimetric>	 but that it was in the current sprint
[10:35:04] <milimetric>	 so I removed it from the sprint (because it seems nobody ever set it to the current sprint)
[10:35:12] <qchris>	 During sprint planning we said that it was there by accident, and we wanted it removed.
[10:35:17] <milimetric>	 ah, good
[10:35:21] <milimetric>	 i found out how to remove it
[10:35:26] <milimetric>	 i logged in as admin and managed the stories
[10:35:27] <qchris>	 Really? How?
[10:35:31] <qchris>	 Ah. Ok.
[10:35:35] <milimetric>	 i can share my admin credential privately if you want them
[10:35:39] <qchris>	 Nono.
[10:35:46] <qchris>	 I think you even already did at some point.
[10:35:57] <milimetric>	 k
[10:36:15] <milimetric>	 as far as hadoop, in this case it might be better if one of you does it
[10:36:25] <qchris>	 Ah ... the bug is gone from the board. Thanks milimetric.
[10:37:01] <milimetric>	 I already did oozie stuff, so others could benefit from the experience
[10:37:10] <milimetric>	 and the other one Andrew said was almost done or something?
[10:37:16] <qchris>	 I can grab one of those tasks soonish.
[10:37:29] <milimetric>	 ok, I'll stop bothering you :)
[10:37:33] <qchris>	 yup, the script is already merged in the repos.
[10:37:46] <qchris>	 It just needs to get run automatically.
[10:40:50] <grrrit-wm>	 (PS1) Milimetric: Update for July meeting [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/143265
[10:41:28] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Update for July meeting [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/143265 (owner: Milimetric)
[10:45:34] <grrrit-wm>	 (CR) Nuria: "Grabbing patch to work on it today." (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[10:49:05] <nuria>	 milimetric: I can certainly work on the hadoop tasks
[10:49:43] <nuria>	 I just need to talk to andrew about the process alarms i set as i cannot tests them due to permits, also would like to finish wikimetrics patch
[10:49:59] <nuria>	 but can grab a hadoop task after that
[10:51:03] <milimetric>	 qchris: sorry about the flip-flop on the backup task, I know we agreed yesterday but looking at it now, it seems scary
[10:51:22] <qchris>	 milimetric: about to chime in on gerrit :-)
[11:07:00] <nuria>	 milimetric, there is still work to do on the "cron user runs reports"
[11:07:16] <nuria>	 as we need to start creating reports for newly register iusers for small wikis
[11:07:55] <nuria>	 qchris, the backup runs with a cron on root user right?
[11:08:33] <nuria>	 so there is nothing additional to do there, correct?
[11:09:03] <qchris>	 nuria: Wikimetrics backup runs as root through cron. Right. I hate it.
[11:10:05] <nuria>	 ok.. ahem.. leaving hate aside..  we said we were leaving it that way or we chnaged that idea?
[11:11:03] <qchris>	 I thought we agreed to leave it like that for now.
[11:11:09] <qchris>	 Didn't we?
[11:11:23] <nuria>	 yes
[11:11:27] <nuria>	 i thought so too
[11:12:38] <nuria>	 then .. what is pending on the task?
[11:14:41] <qchris>	 Like getting it into production, cleaning up the mess around backup someone of our team left behind on staging.
[11:16:57] <milimetric>	 nuria: agreed about "cron user runs reports", that's not done, but I am hopeful that we can finish it this sprint
[11:17:03] <milimetric>	 at least most of the coding should be done
[11:17:43] <nuria>	 ah, i was the one that left a bunch of backed up stuff on staging but i call that testing, not garbage
[11:18:41] <nuria>	 also in dev too , as i tested quite a bit restoring dev from staging backup
[11:18:58] <qchris>	 nuria: Please do not twist my words. I didn't call it "garbage".
[11:19:26] <qchris>	 And what you call "testing" would overwrite production backup :-)
[11:19:35] <qchris>	 Every hour again and again.
[11:19:48] <nuria>	 not in env variables are correct in staging
[11:20:08] <qchris>	 nuria: Thanks for the hint.
[11:20:47] <nuria>	 sorry, "not if env variables are correct in staging"
[11:28:17] <milimetric>	 brb commuting
[11:30:51] <qchris>	 Enjoy the ride, milimetric!
[13:05:10] <grrrit-wm>	 (CR) Gilles: [C: 2] Use 4 spaces to indent, per Python coding standards [analytics/multimedia] - https://gerrit.wikimedia.org/r/143180 (owner: Gergő Tisza)
[13:05:16] <grrrit-wm>	 (Merged) jenkins-bot: Use 4 spaces to indent, per Python coding standards [analytics/multimedia] - https://gerrit.wikimedia.org/r/143180 (owner: Gergő Tisza)
[13:26:23] <ragesoss>	 Is there a faster, more reliable way to pull page view data for large numbers of articles than just sending a whole lot of requests to stats.grok.se json urls?
[13:28:39] <milimetric>	 not yet ragesoss, this is why we're working on the hadoop cluster
[13:29:34] <ragesoss>	 milimetric: thanks. Do you know how glamtools/treeviews does it? Just multiple simultaneous requests?
[13:30:08] <milimetric>	 ragesoss: I believe they have a separate filter that takes out only the data they need to analyze and puts it somewhere for them
[13:30:11] <milimetric>	 but I don't know the specifics
[13:30:32] <milimetric>	 if you're interested, I can ask the people who do
[13:30:48] <milimetric>	 but it probably doesn't apply to your case;  speaking of that, what are you trying to analyze?
[13:30:58] <ragesoss>	 okay, thanks. I tried installing Magnus's code to my own tool account, but it wouldn't pull any pageviews (although the category stuff worked)
[13:31:29] <ragesoss>	 milimetric: my general use case is getting pageview data for the articles worked on by Education Program students.
[13:32:06] <milimetric>	 qchris: what's the difference between PS2 and PS1 here: https://gerrit.wikimedia.org/r/#/c/143268/2
[13:32:33] <ragesoss>	 at the moment, I'm just using a python script to get data for a list of articles that I pulled via SQL query.
[13:32:56] <ragesoss>	 to get a quick-and-dirty baseline for how much reach the work of these students has.
[13:33:18] <qchris>	 milimetric: the wikimetrics submodule's commit hash of the parent.
[13:33:23] <milimetric>	 ragesoss: yeah, I don't know of any better way right now.  But that's *definitely* the kind of questions we're aiming at answering with our projects
[13:33:34] <ragesoss>	 But in the medium term, I expect to be hiring someone to build an automated solution.
[13:33:36] <milimetric>	 1. the growth team is adding campaign-based cohorts
[13:33:42] <milimetric>	 (to wikimetrics)
[13:33:56] <milimetric>	 2. we're standing up the hadoop cluster so we can make a proper queryable source for pageview data
[13:34:10] <ragesoss>	 what's the timeline on the hadoop pageviews?
[13:34:11] <milimetric>	 3. I think we'll put those things together and let you access pageview stats by cohort
[13:34:42] <ragesoss>	 months? a year? two years?
[13:34:57] <milimetric>	 ragesoss: I think that's a better question for tnegrin and kevinator
[13:35:01] <ragesoss>	 :)
[13:35:18] <milimetric>	 but my personal hope is that it's something we get done in 2014
[13:35:51] <ragesoss>	 cool. that's a good piece of info, just have a ballpark of what might be possible.
[13:35:59] <milimetric>	 unlike the past, this is something we are actively prioritizing though and it's no longer on the back burner
[13:37:30] <ottomata>	 nuria, I just checked on hafnium
[13:37:32] <ottomata>	 otto@hafnium:~$ sudo /usr/lib/nagios/plugins/check_eventlogging_jobs
[13:37:32] <ottomata>	 OK: All defined EventLogging jobs are runnning.
[13:37:41] <ottomata>	 mind if I stop the eventlogging jobs and run it?
[13:37:43] <ottomata>	 see what happens?
[13:37:44] <ottomata>	 just to be sure?
[14:00:05] <milimetric>	 qchris: ottomata: standup
[14:00:52] <ottomata>	 thanks
[15:25:50] <nuria>	 milimetric: ottomata & myself are done testing the EL stuff, back to CR-ing wikimetrics
[15:25:58] <milimetric>	 sweet
[15:26:23] <milimetric>	 I'm doing some reading atm, but working on oozie
[15:33:05] <milimetric>	 eventlogging alert
[15:34:08] * milimetric is now an email bot
[15:35:36] <ottomata>	 haha, yeah i know
[15:35:36] <ottomata>	 i'm checking it
[15:36:22] <ottomata>	 good news that it triggered!  i think there's some puppet weirdness that i caused there
[15:36:31] <ottomata>	 but that is good because it actually isn't running right now (nuria :) )
[15:38:36] <grrrit-wm>	 (CR) Nuria: Add pretty symlink for WikimetricsBot (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[15:38:53] <nuria>	 all right!
[15:39:24] <nuria>	 I actually had another bug about this: https://bugzilla.wikimedia.org/show_bug.cgi?id=67309
[15:39:33] <nuria>	 i will add the patchset by hand and close it
[15:43:34] <grrrit-wm>	 (CR) Nuria: [C: 2] "Nice. We certainly do not need this." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/141689 (owner: Milimetric)
[15:44:43] <grrrit-wm>	 (Merged) jenkins-bot: Remove redundant outdated secondary log backup [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/141689 (owner: Milimetric)
[15:46:35] <grrrit-wm>	 (CR) Milimetric: Add pretty symlink for WikimetricsBot (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[15:55:49] <grrrit-wm>	 (CR) Nuria: Add pretty symlink for WikimetricsBot (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[15:59:36] <grrrit-wm>	 (CR) Milimetric: Add pretty symlink for WikimetricsBot (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[16:07:18] <grrrit-wm>	 (PS1) Milimetric: Migrate oozie folder from Kraken minus archive [analytics/refinery] - https://gerrit.wikimedia.org/r/143336 (https://bugzilla.wikimedia.org/67128)
[16:08:04] <milimetric>	 does anyone else get excited when you get like a really cool bugzilla or gerrit number?
[16:08:11] <milimetric>	 143336, score!
[16:15:15] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] "This patch was originally reviewed at:" [analytics/refinery] - https://gerrit.wikimedia.org/r/143336 (https://bugzilla.wikimedia.org/67128) (owner: Milimetric)
[16:33:29] <grrrit-wm>	 (CR) Nuria: Add pretty symlink for WikimetricsBot (2 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/143040 (https://bugzilla.wikimedia.org/66087) (owner: Milimetric)
[16:39:58] <nuria>	 milimetric, qchris need to get groceries, will be back online in 1hr 30 min
[16:40:24] <qchris>	 Happy shopping.
[18:10:08] <nuria>	 i am back ...
[18:14:27] <milimetric>	 i was in a meeting, i'm back too :)
[18:14:38] <milimetric>	 leila! much easier to tab-complete
[18:14:39] <milimetric>	 :)
[18:15:00] <leila>	 totally! :-)
[18:19:24] <nuria>	 after ottomata's help with testing alarms
[18:19:38] <nuria>	 i think the item with EL and graphite counts is done
[18:20:06] <milimetric>	 cool
[18:32:20] <Entalyan>	 qI started the channel-crawIk was ook een beetje door mijn gedult heen. Best. Learning Eperience' ever
[18:36:27] <Entalyan>	 Since I managed to screw up joing the correct channel, would Think the HSenWHo peiopenAlthough it did get me interested... What sort of stuff do you use analplatform Ik was bij ieder station wakker van de PA. Schrok me de tering toen ik 's avonds realatief laat pas thuiskwam) :p
[18:43:07] <kevinator>	 hey milimetric
[18:43:14] <milimetric>	 hi
[18:43:34] <kevinator>	 milimetric: I’m gettinng “Wikimetrics is experiencing problems” message while trying to create a bytes added report
[18:44:27] <grrrit-wm>	 (PS1) Milimetric: Update July pageview data [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/143370
[18:44:40] <milimetric>	 uhoh
[18:44:42] <milimetric>	 checking
[18:45:16] <milimetric>	 kevinator: ?
[18:45:19] <milimetric>	 how / what / where?
[18:45:26] <milimetric>	 i'm not seeing problems, just ran a report
[18:45:36] <kevinator>	 on production… after clicking “Run Report"
[18:45:50] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Update July pageview data [analytics/reportcard/data] - https://gerrit.wikimedia.org/r/143370 (owner: Milimetric)
[18:47:05] <milimetric>	 kevinator: I did that and I got my report, so can you be specific and list the steps / tell me what cohort you're using?
[18:47:19] <kevinator>	 do you want to screenshare?
[18:47:25] <kevinator>	 batcave?
[18:51:40] <kevinator>	 milimetric: it must be my cohort… I can run the same report against other cohorts
[18:53:03] <kevinator>	 milimetric: aha, my cohort has 0 valid users in it
[18:55:09] <milimetric>	 :)
[18:55:17] <milimetric>	 kevinator: that's one of the bugs we've got a patch in for
[18:55:23] <milimetric>	 so it doesn't show that cohort on the report screen
[18:55:40] <kevinator>	 awesome… thanks
[18:55:45] <milimetric>	 kevinator: sorry missed your ping about batcave, I can't hear my pings very well here at the co-working spot
[18:55:56] <kevinator>	 no worries
[18:56:11] <kevinator>	 It made me troubleshoot myself :-)
[19:00:05] <ottomata>	 ack, ori
[19:00:09] <ottomata>	 confine with blocks only works in facter 2
[19:00:19] <ottomata>	 i think...
[19:00:50] <ottomata>	 at least, not working in vagrant...
[19:00:50] <ottomata>	 hm
[19:01:13] <ottomata>	 oops, meant to put that in ops
[19:22:14] <milimetric>	 ottomata: I tried man, I don't think it's physically possible for me to do this task
[19:22:19] <milimetric>	 I'll end up stabbing myself in the face
[19:22:44] <milimetric>	 I remembered that like a year ago I said I would never attempt this again, and I think I was totally right then
[19:22:54] <ottomata>	 haha
[19:22:55] <ottomata>	 cmooon
[19:22:58] <ottomata>	 just talk with meeee!
[19:23:04] <ottomata>	 it is physically possible
[19:23:06] <ottomata>	 what's up?
[19:23:14] <ottomata>	 we can do it together, ya know?
[19:23:17] <nuria>	 "physically possible" ... jaja
[19:23:18] <milimetric>	 :)
[19:23:40] <milimetric>	 i donno man
[19:23:44] <nuria>	 i am happy to grab it tomorrow if you are near the end of your day
[19:23:46] <milimetric>	 all I want to do is write a workflow generator
[19:24:12] <nuria>	 now i have no clue as to how oozie works but hey ...
[19:24:18] <milimetric>	 because no human being should ever have to manually type in all this boiler plate to basically... schedule a job
[19:24:26] <milimetric>	 http://blog.cloudera.com/blog/2013/01/how-to-schedule-recurring-hadoop-jobs-with-apache-oozie/
[19:24:48] <milimetric>	 like, there are three separate files that basically say "every hour do this"
[19:24:58] <milimetric>	 and they're LONG files
[19:25:28] <milimetric>	 with no syntax, grammar, or any way to know what you're doing is right or wrong
[19:26:04] <Entalyan>	 Browsing through the stats and services available through wikimedia.org is fascinating. I never knew you guys published data so openly regarding, well, everything really. Also I like how the domains presented feel completely alien to me because of the nae different type of industry I work in.
[19:26:13] <ottomata>	 there is a pretty specific xml grammer
[19:26:14] <Entalyan>	 Coolest find of the week for me :)
[19:26:19] <ottomata>	 if you wanna really get into it :)
[19:26:23] <milimetric>	 :) yay Entalyan
[19:26:38] <ottomata>	 and, milimetric, the way the webrequest add stuff is structured
[19:26:44] <ottomata>	 you should be able to just copy/paste and change stuff
[19:26:55] <milimetric>	 i've been doing that for a couple of hours
[19:27:13] <milimetric>	 i basically just deleted everything i did in complete confusion
[19:27:37] <ottomata>	 haha, maybe we should do it together?
[19:27:57] <milimetric>	 i don't think it's very useful, honestly, i do appreciate it though :)
[19:28:09] <milimetric>	 not useful because I really plan on never doing this
[19:28:32] <milimetric>	 and if I have to do it, the way i would do it would be to write a wizard that generated the files we need
[19:28:48] <milimetric>	 although, hue seems to have that already
[19:29:03] <milimetric>	 so I'd try that first, but do we have hue running ?
[19:30:49] <milimetric>	 brb, gonna take a walk :)
[19:34:06] <ottomata>	 we don't really have hue running yet, but CDH5 hues seems a little nicer
[19:34:10] <ottomata>	 hopefully will have it more official then
[19:34:14] <ottomata>	 milimetric: when you get back, let's batcave
[20:15:52] <milimetric>	 ottomata: batcave?
[20:18:24] <Entalyan>	 I'm not sure the amount of software is use/development in-house is genious or madness... Impressive regardless. Going to pick that EventLogger extention apart a bit tomorrow to compare the solutions used in subsystems to systems we currently use.
[20:19:03] <Entalyan>	 But I've been saring at unfamiliar code/wiki's for the past 2 hours now, so I'm goikg to give it a rest. Good night!
[21:33:52] <grrrit-wm>	 (PS1) Milimetric: Ignore temporary vim files [analytics/refinery] - https://gerrit.wikimedia.org/r/143485
[21:36:29] <grrrit-wm>	 (PS1) Milimetric: [WIP] Oozify sequence_stats hive script [analytics/refinery] - https://gerrit.wikimedia.org/r/143486 (https://bugzilla.wikimedia.org/67128)
[22:04:07] <grrrit-wm>	 (PS3) Nuria: Fix wiki cohort display for report cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/142514 (owner: Milimetric)
[22:41:47] <grrrit-wm>	 (PS1) Gergő Tisza: [WIP] Track opt-out ratio [analytics/multimedia] - https://gerrit.wikimedia.org/r/143501