[04:40:13] <grrrit-wm>	 (PS2) Terrrydactyl: Added projects to csv output [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/138474
[05:36:37] <grrrit-wm>	 (PS6) AndyRussG: WIP Create a cohort from campaign membership [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/126927 (owner: Awight)
[16:26:28] <Ironholds>	 ottomata, the reducer restarts are still happening, at an earlier stage, even with a restriciton to 10 percent of data
[16:27:11] <grrrit-wm>	 (CR) Nuria: "This is much better. Couple minor points:" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091 (owner: Terrrydactyl)
[16:30:30] <ottomata>	 Ironholds: just curious, what data are you running on?
[16:30:37] <ottomata>	 seeing as uhhh, things are pretty much broken right now :/
[16:31:34] <Ironholds>	 ottomata, define 'broken'?
[16:31:40] <Ironholds>	 and webrequest_source='text'
[16:31:43] <Ironholds>	 data from June
[16:31:47] <ottomata>	 well, new data since saturday
[16:31:50] <ottomata>	 not coming in 100%
[16:31:56] <ottomata>	 we ahve a broker down, which means lots of dropped messages
[16:32:07] <ottomata>	 i am in the process of taking one datanode out of the cluster, then we will take a couple more probably
[16:32:08] <Ironholds>	 yee
[16:32:10] <Ironholds>	 kk
[16:32:24] <Ironholds>	 so, with this query we have a reducer die-off at 4% and then the same pattern as before
[16:32:28] <ottomata>	 aye
[16:32:29] <Ironholds>	 33%, stalling, stalling, 40%, die.
[16:32:41] <ottomata>	 Ironholds: i'm really sorry I haven't had time to look into your queries since last week really :/
[16:32:44] <Ironholds>	 There shouldn't be any consistency in which machines are assigned in which order, should they?
[16:32:46] <Ironholds>	 it's okay.
[16:32:52] <ottomata>	 i have to do these stupid reviews today, and we have more meetings
[16:32:52] <ottomata>	 sigh
[16:34:51] <ori>	 ottomata: you're doing awesome work, the pieces will align, don't worry
[16:35:26] <ottomata>	 thanks, ori :)
[16:35:27] <ori>	 i'm at some remove and from my vantage point things look like they're coming together
[16:35:41] <ottomata>	 yeah, i think so too, just this past weekend wasn't a good sign :/
[16:35:50] <ottomata>	 i mean, we knew something like that could happen, hence the planned capacity increases
[16:35:50] <ottomata>	 but ja
[16:36:01] <ottomata>	 also, Ironholds has crazy query problems that are hard to troubleshoot
[16:36:19] <ottomata>	 that will either be solved by me tweaking some yet unknown hadoop knobs
[16:36:24] <ottomata>	 or understanding how to make his queries better
[16:37:36] <Ironholds>	 yeah, I have no idea what's going on
[16:37:47] <Ironholds>	 but I'm happy I could break everything *before* we had production systems on it :D
[17:03:42] <jwild>	 hello everyone! are we having a showcase today?
[18:01:25] <grrrit-wm>	 (PS9) Terrrydactyl: Add ability to tag a cohort [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/133091
[18:29:54] <qchris_meeting>	 milimetric: Google does not allow me to reconnect.
[18:30:05] <qchris_meeting>	 milimetric: Might take a few minutes :-(
[20:42:01] <grrrit-wm>	 (PS1) QChris: Make daily dammit compact script rsync files if monthly step fails [analytics/wikistats] - https://gerrit.wikimedia.org/r/138685