[00:10:03] <grrrit-wm>	 (PS1) QChris: Create empty CSVs for relevant wikis [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172181
[00:10:05] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts for 2008 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172182
[00:11:23] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts for 2009 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172183
[00:12:14] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts for 2010 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172185
[00:13:14] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts for 2011 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172186
[00:14:31] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts for 2012 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172187
[00:15:54] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts for 2013 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172188
[00:17:31] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts for 2014 up to 2014-09-22 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172189
[00:19:20] <grrrit-wm>	 (PS1) QChris: Backfill daily projectcounts up to 2014-11-08 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172190
[02:10:43] <wikibugs>	 Analytics / EventLogging: Event Logging tables that we can drop as of Nov 7th - https://bugzilla.wikimedia.org/73140#c1 (Sean Pringle) NEW>RESO/FIX Done.
[03:03:58] <grrrit-wm>	 (PS5) Nuria: Add Separated Values (CSV, TVS) data converter [analytics/dashiki] - https://gerrit.wikimedia.org/r/168488 (owner: Milimetric)
[04:03:46] <grrrit-wm>	 (PS1) QChris: Add basic python setup [analytics/aggregator] - https://gerrit.wikimedia.org/r/172194 (https://bugzilla.wikimedia.org/72740)
[04:03:51] <grrrit-wm>	 (PS1) QChris: Add basic implementation of projectcount aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740)
[04:03:55] <grrrit-wm>	 (PS1) QChris: Add basic monitoring script for projectcount aggregates [analytics/aggregator] - https://gerrit.wikimedia.org/r/172196 (https://bugzilla.wikimedia.org/72740)
[04:04:00] <grrrit-wm>	 (PS1) QChris: Allow additional logging to disk for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172197 (https://bugzilla.wikimedia.org/72740)
[04:04:05] <grrrit-wm>	 (PS1) QChris: Add switch for automatic pushing of data repo for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172198 (https://bugzilla.wikimedia.org/72740)
[04:08:26] <grrrit-wm>	 (PS2) QChris: Create empty CSVs for relevant wikis [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172181 (https://bugzilla.wikimedia.org/72740)
[04:08:30] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts for 2008 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172182 (https://bugzilla.wikimedia.org/72740)
[04:08:35] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts for 2009 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172183 (https://bugzilla.wikimedia.org/72740)
[04:11:04] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts for 2010 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172185 (https://bugzilla.wikimedia.org/72740)
[04:11:09] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts for 2011 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172186 (https://bugzilla.wikimedia.org/72740)
[04:11:14] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts for 2012 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172187 (https://bugzilla.wikimedia.org/72740)
[04:14:25] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts for 2013 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172188 (https://bugzilla.wikimedia.org/72740)
[04:14:30] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts for 2014 up to 2014-09-22 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172189 (https://bugzilla.wikimedia.org/72740)
[04:14:35] <grrrit-wm>	 (PS2) QChris: Backfill daily projectcounts up to 2014-11-08 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172190 (https://bugzilla.wikimedia.org/72740)
[13:04:12] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts for 2008 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172182 (https://bugzilla.wikimedia.org/72740)
[13:04:14] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts for 2009 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172183 (https://bugzilla.wikimedia.org/72740)
[13:04:16] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts for 2010 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172185 (https://bugzilla.wikimedia.org/72740)
[13:08:15] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts for 2011 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172186 (https://bugzilla.wikimedia.org/72740)
[13:08:17] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts for 2012 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172187 (https://bugzilla.wikimedia.org/72740)
[13:08:19] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts for 2013 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172188 (https://bugzilla.wikimedia.org/72740)
[13:11:07] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts for 2014 up to 2014-09-22 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172189 (https://bugzilla.wikimedia.org/72740)
[13:11:09] <grrrit-wm>	 (PS3) QChris: Backfill daily projectcounts up to 2014-11-08 [analytics/aggregator/data] - https://gerrit.wikimedia.org/r/172190 (https://bugzilla.wikimedia.org/72740)
[13:14:00] <grrrit-wm>	 (PS1) QChris: Add header line to CSVs for projectcounts aggregation [analytics/aggregator] - https://gerrit.wikimedia.org/r/172234
[13:14:02] <grrrit-wm>	 (PS1) QChris: Switch to CRLF as line terminator for CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172235
[13:14:04] <grrrit-wm>	 (PS1) QChris: Allow to force recomputation of existing data [analytics/aggregator] - https://gerrit.wikimedia.org/r/172236
[13:14:06] <grrrit-wm>	 (PS1) QChris: Add total column for project aggregation CSVs [analytics/aggregator] - https://gerrit.wikimedia.org/r/172237
[13:14:08] <grrrit-wm>	 (PS1) QChris: Add bad dates for projectcount aggregations [analytics/aggregator] - https://gerrit.wikimedia.org/r/172238
[13:14:34] <grrrit-wm>	 (Abandoned) QChris: Add bad dates for projectcount aggregations [analytics/aggregator] - https://gerrit.wikimedia.org/r/172238 (owner: QChris)
[15:01:11] <wikibugs>	 Analytics / Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://bugzilla.wikimedia.org/69615#c21 (christian) Happened again for:    2014-11-09T21/2H (on upload)
[15:01:29] <qchris_meeting>	 !log Marked raw upload webrequest partition for 2014-11-09T22/1H ok (See {{bug|69615#c21}})
[15:03:06] <ottomata>	 AHH
[15:03:17] <ottomata>	 having google trouble...
[15:18:28] <grrrit-wm>	 (PS1) Gilles: Update mysql command to use shared config file [analytics/multimedia] - https://gerrit.wikimedia.org/r/172252
[15:18:51] <grrrit-wm>	 (CR) Gilles: [C: 2] "Command tested during ssh session" [analytics/multimedia] - https://gerrit.wikimedia.org/r/172252 (owner: Gilles)
[15:18:56] <grrrit-wm>	 (Merged) jenkins-bot: Update mysql command to use shared config file [analytics/multimedia] - https://gerrit.wikimedia.org/r/172252 (owner: Gilles)
[15:52:22] <ottomata>	 oo, hey, there's a thread on the kafka mailing list about adding people to the Power By page
[15:52:26] <ottomata>	 https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
[15:52:41] <ottomata>	 I'm going to send them something.  think I should say anything more than this?
[15:53:20] <ottomata>	 The Wikimedia Foundation uses Kafka as a log transport for analytics data from production webservers and applications.  This data is consumed into Hadoop using Camus and to processors of analytics data.
[15:53:24] <ottomata>	 meh, needs edits ^help!
[15:53:29] <ottomata>	 (qchris? ^)
[15:53:42] * qchris just loaded that page
[15:54:14] <ottomata>	 other
[15:54:15] <qchris>	 That's fine.
[15:54:22] <ottomata>	 The Wikimedia Foundation uses Kafka as a log transport for analytics data from production webservers and applications.  This data is consumed into Hadoop using Camus and to other processors of analytics data.
[15:54:32] <ottomata>	 hokay!
[15:59:25] <wikibugs>	 Analytics / Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://bugzilla.wikimedia.org/69615#c22 (christian) Happened again for:    2014-11-10T10/2H (on upload)
[16:00:20] <qchris>	 !log Marked raw upload webrequest partition for 2014-11-10T10/2H ok (See {{bug|69615#c22}})
[16:08:17] <Ironholds>	 mforns, who wants a present? ;p
[16:08:29] <mforns>	 :]
[16:08:46] <mforns>	 you have it
[16:08:47] <mforns>	 ?
[16:09:51] <Ironholds>	 I have it! Or will in 30 seconds when I've aggregated to the month level as Toby requested.
[16:09:55] <Ironholds>	 I think. It was month, right?
[16:10:54] <mforns>	 yes, but if you have it like on friday and want me to do it with a select, I already have the query
[16:11:15] <mforns>	 Ironholds: ^
[16:11:21] <Ironholds>	 ahhh
[16:11:30] <Ironholds>	 eh, easiest at my end. 'angon.
[16:12:46] <mforns>	 ok, so where is it? the same place? what table?
[16:14:30] <grrrit-wm>	 (PS1) Ottomata: Fix user in kafkatee.logrotate [analytics/kafkatee] - https://gerrit.wikimedia.org/r/172262
[16:14:41] <grrrit-wm>	 (Abandoned) Ottomata: Fix user in kafkatee.logrotate [analytics/kafkatee] - https://gerrit.wikimedia.org/r/172262 (owner: Ottomata)
[16:15:09] <grrrit-wm>	 (PS1) Ottomata: Fix user in kafkatee.logrotate [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/172263
[16:15:30] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Fix user in kafkatee.logrotate [analytics/kafkatee] (debian) - https://gerrit.wikimedia.org/r/172263 (owner: Ottomata)
[17:09:00] <Ironholds>	 mforns_lunch, almost done!
[17:24:25] <Ironholds>	 mforns_lunch, check out staging.pentahoviews
[18:00:52] <grrrit-wm>	 (PS1) Gilles: Output script actually doesn't need the table in the SQL command [analytics/multimedia] - https://gerrit.wikimedia.org/r/172291
[18:01:13] <grrrit-wm>	 (CR) Gilles: [C: 2] Output script actually doesn't need the table in the SQL command [analytics/multimedia] - https://gerrit.wikimedia.org/r/172291 (owner: Gilles)
[18:01:19] <grrrit-wm>	 (Merged) jenkins-bot: Output script actually doesn't need the table in the SQL command [analytics/multimedia] - https://gerrit.wikimedia.org/r/172291 (owner: Gilles)
[18:01:24] <YuviPanda>	 ottomata: hmm, jdlrobson tried sudo again for the research password.
[18:06:39] <ottomata>	 ARGH
[18:06:56] <ottomata>	 YuviPanda: i am working with a new computer today and my ssh key and my puppet repo are in a weird state
[18:07:09] <YuviPanda>	 ottomata: want me to add him?
[18:07:15] <ottomata>	 yeah, add him to resaerchers group
[18:07:21] <ottomata>	 do you have merge rights?
[18:07:47] <Ironholds>	 YuviPanda, why would he do that?
[18:07:56] <ottomata>	 because he is not in the researchers grou
[18:07:57] <ottomata>	 p
[18:08:02] <YuviPanda>	 ottomata: I do! :)
[18:08:06] <Ironholds>	 yes, but if you have to sudo to get permissions to something...
[18:08:12] <ottomata>	 he can't sudo
[18:08:14] <ottomata>	 he is just trying
[18:08:15] <Ironholds>	 you should probably not be accessing it.
[18:08:16] <Ironholds>	 that too
[18:08:19] <YuviPanda>	 indeed.
[18:08:30] <Ironholds>	 YuviPanda, do me a favour; when you land in SF, give him a look of disapproval? ;p
[18:08:55] <YuviPanda>	 Ironholds: doing similar things got me and milimetric a sternly worded letter from leslie a few years back :)
[18:09:05] <Ironholds>	 yup
[18:09:15] <Ironholds>	 I accidentally once sudod and got told off by, I think, Tim
[18:09:26] <Ironholds>	 (I forgot which terminal window was to my local and which was to stat1
[18:09:33] <YuviPanda>	 yeah, I'm wondering if I should send him an email with a link to https://xkcd.com/838/
[18:09:39] <YuviPanda>	 oh I had no excuse, I was just exploring :)
[18:09:55] <YuviPanda>	 ottomata: +1? https://gerrit.wikimedia.org/r/#/c/172292/
[18:10:15] <Ironholds>	 YuviPanda, do it
[18:10:16] <Ironholds>	 dooo iiiit
[18:10:54] <ottomata>	 merge away, thank you
[18:11:13] <YuviPanda>	 ottomata: cool, puppetmerging shortly.
[18:11:23] <ottomata>	 goign to eat lunch before ops meetinb, bbl
[18:16:47] <YuviPanda>	 Ironholds: done
[18:17:01] <Ironholds>	 ta
[18:31:03] <mforns>	 Ironholds: Thanks!
[18:31:10] <Ironholds>	 np!
[19:03:53] <Ironholds>	 mforns, any luck?
[19:08:21] <mforns>	 Ironholds: haven't started yet, i'm finishing a CR, will get it in a sec
[19:08:46] <Ironholds>	 kk
[19:26:30] <nuria__>	 qchris: still working on pageviews have not forgotten but there are some major spikes in the data, I think, lemme do some plots
[19:27:03] <qchris>	 Yes, there are spikes
[19:27:15] <qchris>	 there's not much we can do about it.
[19:27:31] <qchris>	 I noted some issues in the commit messages of
[19:27:59] <qchris>	 https://gerrit.wikimedia.org/r/#/projects/analytics/aggregator/data,dashboards/default:recent
[19:32:20] <grrrit-wm>	 (CR) Mforns: "I think everything was fine!" (6 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/168488 (owner: Milimetric)
[19:33:00] <mforns>	 Ironholds: starting now
[19:37:37] <Ironholds>	 ta
[19:38:21] <mforns>	 Ironholds: you speak portuguese?
[19:38:50] <Ironholds>	 mforns, no! As a side-effect of there only being a limited number of sounds, "ta" is also British English for "thanks :)
[19:38:52] <Ironholds>	 *"
[19:39:03] <mforns>	 xD
[19:39:08] <Ironholds>	 I didn't know of its portugese meaning (googled!)
[19:39:25] <mforns>	 in portuguese tá is short for está
[19:39:27] <mforns>	 oh! ok
[20:14:12] <nuria__>	 qchris: ok, did some plots and spikes are not as bad as vega shows them soooo. something is going on
[20:14:43] <qchris>	 nuria__: Not sure what you mean. Can you point me to a vega rendering?
[20:15:02] <nuria__>	 qchris: no, not outside of mydesktop right now
[20:15:08] <qchris>	 Screenshot?
[20:15:14] <nuria__>	 qchris: will investigate a bit
[20:15:26] <qchris>	 Ok. That works too :-)
[20:15:31] <qchris>	 Ping me if you need help.
[20:21:18] <YuviPanda>	 do point questions about research pw to me today :)
[20:22:19] <tnegrin>	 hi mforns
[20:22:29] <mforns>	 hi tnegrin
[20:22:36] <tnegrin>	 no pressure, just wondering about the ETA of the Pentaho stuff
[20:23:08] <mforns>	 I think it will take like 2 hours
[20:23:33] <mforns>	 is this OK?
[20:24:11] <tnegrin>	 sure -- can you ping me when it's ready? it's a holiday here tomorrow so we want to look at it today if possible
[20:24:48] <mforns>	 of course
[20:25:08] <mforns>	 I'll ping you in short tnegrin
[20:25:14] <tnegrin>	 thanks marcel!
[20:25:19] <mforns>	 np
[20:55:56] <YuviPanda>	 ottomata: halfak hmm, do we use mongodb at all in stat1003?
[20:56:05] <YuviPanda>	 I think puppet has it set to ensure that the service is stopped
[20:56:07] <YuviPanda>	 perhaps we could kill it
[20:56:17] * halfak is not here
[20:56:21] <YuviPanda>	 ok
[20:56:23] <halfak>	 Also, I don't use mongodb on stat3 anymore
[20:56:24] <YuviPanda>	 :)
[20:56:25] <halfak>	 :)
[20:56:28] <YuviPanda>	 alright
[21:01:03] <ottomata>	 YuviPanda: I think halfak was the only one who used it, not sure
[21:01:13] <YuviPanda>	 yeah, that's my impression too
[21:01:25] <grrrit-wm>	 (CR) Hashar: [C: 1] "recheck" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172194 (https://bugzilla.wikimedia.org/72740) (owner: QChris)
[21:06:32] <grrrit-wm>	 (CR) Hashar: "Before merging this, one probably want to have the *flake8-bin job to be triggered by Zuul." [analytics/aggregator] - https://gerrit.wikimedia.org/r/172195 (https://bugzilla.wikimedia.org/72740) (owner: QChris)
[21:08:21] <grrrit-wm>	 (CR) Hashar: "That is all good to me. I would merge it myself but I guess you probably review by other members of the analytics team :-]" [analytics/aggregator] - https://gerrit.wikimedia.org/r/172194 (https://bugzilla.wikimedia.org/72740) (owner: QChris)
[21:22:23] <qchris>	 ottomata: Could I bribe you to have a quick look on the projectcounts puppet stuff to see if I should improve anything?
[21:22:48] <qchris>	 (No need to merge, as the python code has not been reviewed ... just to have puppet parts ready once the python code got reviewed)
[21:23:16] <ottomata>	 will do, gimme just a few
[21:23:22] <qchris>	 Awesome! Thanks.
[21:28:03] <Ironholds>	 ottomata, what is the "stats" account and why is it sucking up everything stat1002 has?
[21:28:25] * YuviPanda checks
[21:28:47] <ottomata>	 YuviPanda:  is your man for the moment,, Ironholds, I am having unexpected trouble using my new laptop
[21:57:57] <Ironholds>	 ottomata, a friend suggests having sshd automatically at the highest niceness value (or, lowest?) to avoid this problem in the future. Not sure if that'd be useful? (not familiar with how we currently handle that)
[21:58:38] <ottomata>	 probably not a bad idea
[21:58:48] <ottomata>	 qchris: ok, review easier via irc
[21:58:52] <ottomata>	 how about
[21:59:04] <ottomata>	 make your base class just called
[21:59:04] <ottomata>	 misc::statistics::aggregator
[21:59:06] <ottomata>	 and then
[21:59:31] <ottomata>	 don't set $data_path there, as it is the only thing that has anything to do with 'projectcounts' in that path
[21:59:34] <ottomata>	 then
[21:59:38] <ottomata>	 in your other class
[21:59:44] <ottomata>	 you can inherit from misc::statistics::aggregator
[21:59:54] <ottomata>	 and have all the variables from that class in local scope
[22:00:01] <ottomata>	 then you can set $data path there
[22:00:05] <ottomata>	 i guess your subclass could be
[22:00:15] <ottomata>	 class misc::statistics::aggregator::pagecounts
[22:00:38] <ottomata>	 as all it has to do is make sure that the $data_path exists, and that hte cron job for aggregated projectounts runs
[22:00:59] <qchris>	 Sorry. read up on stat1002 and it's uptime :-)
[22:01:22] <YuviPanda>	 tch tch misc :)
[22:02:06] <qchris>	 ottomata: k. thanks.
[22:02:14] <qchris>	 What about the /mnt/hdfs ...
[22:02:17] <ottomata>	 YuviPanda: I don't want to talk about it :/
[22:02:20] <ottomata>	 :)
[22:02:20] <qchris>	 Is that ok to hardcode?
[22:02:23] <YuviPanda>	 hehe :)
[22:02:31] <ottomata>	 ah, qchris, hm, its ok, but yeah, it would be better to refernce
[22:02:38] <YuviPanda>	  /srvvvvv!!! :)
[22:02:40] <ottomata>	 but, probably not that great to include it, so, you can just set up the dep without including!
[22:02:41] * YuviPanda stops fly by reviewing
[22:03:06] <ottomata>	 at the top of your ::projectcounts class (because that is the one that needs the mount)
[22:03:06] <ottomata>	 do
[22:03:31] <ottomata>	 Class['cdh::hadoop::mount'] -> Class['misc::statistics::aggregator::projectcounts]
[22:03:40] <qchris>	 Ok.
[22:03:41] <qchris>	 Thanks.
[22:03:52] <ottomata>	 that will set up an explicit dependency between the classes, but won't actually attempt to include the class
[22:04:04] <ottomata>	 basically, puppet will fail if that class is not included elsewhere
[22:04:40] <qchris>	 It's better than it is now :-) Great!
[22:04:40] <ottomata>	 and, qchris, as for cron time, any time is fine.
[22:04:45] <qchris>	 k.
[22:05:26] <qchris>	 And any comments on the "mount aggregated project counts files in wikimetrics' public directory" change?
[22:06:00] <qchris>	 https://gerrit.wikimedia.org/r/#/c/172285/
[22:06:01] <ottomata>	 lnk?
[22:06:01] <ottomata>	 looking
[22:06:01] <ottomata>	 link*
[22:06:13] <hashar>	 qchris: I have merged a flake8 Jenkins job for analytics/aggregator
[22:06:23] <hashar>	 qchris: flake8-bin is not triggered yet though
[22:06:38] <qchris>	 hashar: Yes, I saw it and commented.
[22:06:47] <qchris>	 The recheck worked :-)
[22:07:04] <qchris>	 Triggering is at https://gerrit.wikimedia.org/r/#/c/172399/
[22:07:36] <qchris>	 Thanks for the quick merge of the first part :-)
[22:07:59] <ottomata>	 qchris: yes, you have the /datafiles say ensure => directory and there is a target
[22:08:12] <hashar>	 qchris: nice :-]
[22:08:23] <hashar>	 qchris:  I would have merged the change introducing tox.ini but refrained
[22:08:23] <ottomata>	 Also, you don't need to enclose your require in an array [ ] if there is only one require
[22:08:41] <qchris>	 hashar: I read that comment ... you're joining us for standup .. you're part of the team :-)
[22:08:57] <qchris>	 hashar: But I'll get analytics people to review it. Thanks.
[22:09:02] <ottomata>	 qchris: , will put comments in that change instead of here, that one is easier
[22:09:11] <qchris>	 ottomata: k. thanks.
[22:17:20] <hashar>	 qchris: thank you!
[22:18:40] <qchris>	 hashar: Thanks for making continuous integration so easy! It's the first job I set up for analytics ... and it is sooooo easy.
[22:18:45] <qchris>	 It's amazing!
[22:18:58] <qchris>	 And there he goes :-) Hahaha
[22:31:27] <wikibugs>	 Analytics / General/Unknown: Operationalize Saiku - https://bugzilla.wikimedia.org/73246 (Toby Negrin) NEW p:Unprio s:normal a:None We like this visualization platform. What would it take to put it in production?
[22:34:19] <Ironholds>	 hey qchris/ottomata, did we do anything to the sampled logs in 2013-10?
[22:34:27] <Ironholds>	 there's this TREMENDOUS jump in pageviews
[22:34:39] <Ironholds>	 4bn to 15bn. What did we do differently?
[22:34:56] <qchris>	 Was'n that CentralAuth?
[22:35:12] <Ironholds>	 naw, Special pages are being filtered
[22:35:30] <Ironholds>	 hmn. Looks like it hit the desktop site specifically
[22:35:32] * Ironholds goes down another level
[22:35:47] <nuria__>	 Ironholds: actually look per project
[22:36:05] <Ironholds>	 good idea
[22:36:08] <qchris>	 Google referers changed a lot during that time.
[22:36:13] <nuria__>	 in enwiki things kind of make sense over the timeperiod (desktop, mobile)
[22:36:36] <qchris>	 They jumped up sometime in autumn 2013.
[22:36:52] <qchris>	 Meh ... but not 11bn :-)
[22:36:55] <Ironholds>	 yeah
[22:37:03] <nuria__>	 but in other projects (es.mobile) if you plot pageviews in the time period it jumps arround too much
[22:37:04] <Ironholds>	 if we had 4bn PVs prior to that we'd be in twubble
[22:37:22] <nuria__>	 from 1 million a month to 100.000 a month a few months later
[22:37:30] <qchris>	 Ironholds: It's hard to tell without knowing what you filter / don't filter. Sorry.
[22:37:47] <Ironholds>	 yeah, I'll dig in and see what I find if I run it over two different days
[22:38:09] <qchris>	 nuria__: Where should the aggregated csvs go in the wikimetrics' public directory?
[22:38:13] <qchris>	 Something like "http://metrics.wmflabs.org/public/projectcounts/daily"
[22:38:14] <nuria__>	 Ironholds: seems like the data  2013-10 to 2013-11 cannot be compared
[22:38:15] <Ironholds>	 I /have/ discovered that my URL-parsing solution is failing. Womp womp.
[22:38:16] <qchris>	 or "http://metrics.wmflabs.org/public/datafiles/DailyPageviews"
[22:38:20] <qchris>	 or something else?
[22:38:29] <Ironholds>	 nuria__, well, I'll see what I find
[22:38:39] <Ironholds>	 if it turns out to be some historical artefact, I can easily run over 6 months of logs by wednesday
[22:38:51] <Ironholds>	 if it's something from 2013-11 /onwards/ we're in twubble
[22:39:02] <nuria__>	 qchris: https://metrics.wmflabs.org/static/public/datafiles/<daily-something>
[22:39:22] <nuria__>	 qchris: <daily-something> _. name of metric
[22:39:44] <qchris>	 nuria__: So since metrics are currently camelcased ...
[22:39:53] <qchris>	 Does "http://metrics.wmflabs.org/public/datafiles/DailyPageviews" look right?
[22:40:04] <nuria__>	 sure, that is how i have it
[22:40:07] <nuria__>	 right now
[22:40:10] <qchris>	 Cool.
[22:40:17] <nuria__>	 qchris: brb, need to go pick my daughter
[22:40:23] <qchris>	 Sure.
[22:40:32] <Ironholds>	 per-project isn't getting me anything
[22:40:48] <Ironholds>	 I'm going to go into the hourly-level resolution sets and see what I can find there
[22:40:55] <Ironholds>	 see if I can pick out a particular day/hour that dramatically changed.
[22:40:58] <Ironholds>	 should narrow down the hunting
[22:42:06] <Ironholds>	 tnegrin, for your reading pleasure: massive unexplained jump in PVs in my data from 2013-10 to 2013-11 (like, fourfold increase)
[22:42:06] <qchris>	 Ironholds: With that increase, one would assume that the TSVs size dramatically increased. But it did not.
[22:42:08] <Ironholds>	 tracking down as to why.
[22:42:20] <Ironholds>	 qchris, indeed. Alternately that the things feeding into it were different?
[22:42:29] <Ironholds>	 I'll go through and see.
[22:42:33] <tnegrin>	 huh
[22:43:56] <Ironholds>	 pulling out hourly-level aggregates to try and narrow it down
[22:44:09] <Ironholds>	 I've also discovered my project aggregator didn't work for historical wikisource URLs. will track down.
[22:45:01] <mforns>	 Ironholds, qchris: The TSV file has one line for each combination of dimensions, this should not necessarily grow with the increase in the value
[22:45:57] <Ironholds>	 mforns, no, the sampled logs this is based on
[22:46:14] <qchris>	 mforns: I was referring to the sampled-1000 TSVs on stat1002 at /a/squid/archive/sampled/sampled-1000.tsv.log-*
[22:46:24] <mforns>	 oh! sorry
[22:47:32] <tnegrin>	 Ironholds: the data looks ok after 11/13
[22:48:04] <Ironholds>	 yup
[22:48:10] <Ironholds>	 8 November 2013 mean anything to anyone?
[23:05:28] <Nemo_bis>	 Is http://datasets.wikimedia.org/public-datasets/all/registrations/daily_reg_mobile.csv supposed to be empty?
[23:18:43] <tnegrin>	 hey Ironholds -- what's going on?
[23:18:59] <Ironholds>	 tnegrin, I'm digging into the datasets trying to work out where the drop-off happens and wh
[23:18:59] <Ironholds>	 y
[23:19:09] <Ironholds>	 pinned it down to starting around 8 November
[23:19:38] <tnegrin>	 the data after that seems fine
[23:19:59] <Ironholds>	 yup
[23:20:09] <Ironholds>	 so I'm grabbing 1 day from before and 1 day from after and seeing what the difference is
[23:20:52] <tnegrin>	 how about we just do december through october in the cube?
[23:21:19] <Ironholds>	 sure
[23:21:30] <Ironholds>	 but if I can work this out we can get at least a year, so I'd like to at least try.
[23:21:44] <Ironholds>	 oh wow, that IS a drop
[23:21:45] <Ironholds>	 huh
[23:21:57] <tnegrin>	 we need to hurry
[23:21:57] <Ironholds>	 after all of filtering it should be left with 600k hits
[23:21:58] <Ironholds>	 not 185524
[23:22:00] <Ironholds>	 yes, I know.
[23:22:04] <tnegrin>	 sorry dude
[23:22:09] <Ironholds>	 so, go with December onwards for now, and I'll see if I can debug
[23:22:20] <tnegrin>	 marcel might need to sleep
[23:22:28] <Ironholds>	 if I can debug in time to run for the remaining time period - which will not take long in runtime terms, it's only around 6 months - great
[23:22:39] <Ironholds>	 and then marcel can put it in tomorrow and we are go for wednesday
[23:22:43] <Ironholds>	 otherwise, we have 11 months of data
[23:22:46] <Ironholds>	 which is still really good
[23:23:52] <mforns>	 tnegrin, no no, here it's fine
[23:24:06] <mforns>	 I will stay here for 1h
[23:24:19] <tnegrin>	 ok -- thanks mforns
[23:24:55] <DarTar>	 for a half a second I misread > Ironholds: “qchris: brb, need to go pick my daughter “
[23:25:25] <Ironholds>	 mforns, well, we won't have new data in an hour.
[23:25:37] <Ironholds>	 not without breaking stat1002 and I leave that to other people *looks at ellery ;p*
[23:25:55] <Ironholds>	 DarTar, that's...yeah: not going to be a sentence I utter, I think.
[23:29:12] <mforns>	 Ironholds, OK
[23:30:55] <Ironholds>	 FOUND IT.
[23:31:00] <Ironholds>	 ...huh. this is the weirdest thi-OH FUCK.
[23:31:03] * Ironholds headdesks
[23:31:51] <mforns>	 what was it?
[23:32:09] <Ironholds>	 checking, checking...
[23:32:25] <Ironholds>	 that is the /dumbest thing/
[23:32:32] <Ironholds>	 so: none of the PV filters were killing anything wrong
[23:32:48] <Ironholds>	 but, there is a filter to remove rows that are malformed before they do something stupid with future processing
[23:33:00] <Ironholds>	 one great indicator as to whether it is malformed is whether the timestamp follows the right format
[23:33:12] <Ironholds>	 2013-11 was when we stopped storing Y-M-D-H-M-S-ms
[23:33:33] <Ironholds>	 and of course my strptime function is designed to handle YYYY-MM-DDTHH:MM:SS because that's all we've had for the last year.
[23:34:20] * Ironholds headslaps
[23:34:23] <Ironholds>	 okay, can patch
[23:34:26] <mforns>	 oh...
[23:34:51] <mforns>	 fine!
[23:35:28] <Ironholds>	 but, this will be done tomorrow.
[23:35:40] <Ironholds>	 go sleep in the meantime
[23:42:47] <Ironholds>	 tnegrin, fixed the bug upstream
[23:42:52] <Ironholds>	 we're good. Gonna rerun for those months.
[23:42:59] <tnegrin>	 ok
[23:49:07] <tnegrin>	 mforns: looks like we can stand down
[23:49:17] <DarTar>	 Ironholds: neglect my last mail, Toby told me
[23:49:53] <mforns>	 tnegrin, OK
[23:49:54] <Ironholds>	 kk
[23:50:23] <tnegrin>	 thanks for the help -- I think Ironholds will be done in 24 hours or so and we can reload the data on Wednesday
[23:51:25] <tnegrin>	 Ironholds: to confirm -- the data in the existing series should be accurate b/t dec of 2013 and october 2014?
[23:52:00] <Ironholds>	 yup
[23:52:05] <DarTar>	 Ironholds: awesome
[23:52:10] <Ironholds>	 that's appears to be when the remaining machines transitioned their TS format
[23:52:35] <mforns>	 Ironholds, you're going to work tomorrow?
[23:52:41] <DarTar>	 even just having a couple of accurate months in 2014 can help us answer a number of questions
[23:52:54] <Ironholds>	 mforns, I'm going to carefully monitor my scripts
[23:53:01] <DarTar>	 speaking of which, Ironholds: since you asked about new dimensions we may want to add to the cube, guess what the elephant is the room is
[23:53:02] <Ironholds>	 and then I'm going to check my paper for WWW got submitted and works
[23:53:03] <mforns>	 ok, I see
[23:53:05] <Ironholds>	 and then I'm going to go drinking
[23:53:09] <Ironholds>	 DarTar, oh god. what?
[23:53:11] <mforns>	 xD
[23:53:45] <DarTar>	 Ironholds: the single biggest change impacting our traffic that we rolled out in summer 2013 and we haven’t seriously monitored ever since
[23:53:52] <Ironholds>	 SSH usage?
[23:53:55] <Ironholds>	 *HTTPS
[23:54:00] <DarTar>	 bingo
[23:54:05] <Ironholds>	 I'll see if the x_analytics flags for that exist across all time
[23:55:24] <DarTar>	 Ironholds: I imagine that having a boolean isSSL dimension combined with geographic data *over time* will allow us to understand a lot more about what’s happening with non-spider/non-robot PV trends
[23:55:42] <DarTar>	 sorry, the last one was non-spider/non-automated
[23:56:17] <Ironholds>	 sure, if the field is intact. we'll see.
[23:56:18] <DarTar>	 Ironholds: let’s add this to the wishlist for after Nov 17 ;)
[23:56:25] <Ironholds>	 I'm seeing weird stuff happen with the mobile versions
[23:56:26] <DarTar>	 Ironholds: right
[23:56:54] <DarTar>	 meaning?
[23:56:59] <DarTar>	 note that this will be superuseful to sanity check the referred traffic data too