[08:11:48] <grrrit-wm1>	 (CR) Gilles: [C: 2] Fix global optout counts [analytics/multimedia] - https://gerrit.wikimedia.org/r/155272 (owner: Gergő Tisza)
[08:11:56] <grrrit-wm1>	 (Merged) jenkins-bot: Fix global optout counts [analytics/multimedia] - https://gerrit.wikimedia.org/r/155272 (owner: Gergő Tisza)
[09:30:19] <wikibugs>	 Analytics / Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://bugzilla.wikimedia.org/69615#c5 (christian) Happened again for:    2014-08-23T16:xx:xx/2014-08-23T17:xx:xx (on upload)
[10:12:18] <wikibugs>	 Analytics / General/Unknown: Raw webrequest partitions for 2014-08-23T20:xx:xx not marked successful - https://bugzilla.wikimedia.org/69971 (christian) NEW p:Unprio s:normal a:None For the hour 2014-08-23T20:xx:xx, none [1] of the the four sources' bucket was marked successful.  What happen...
[10:16:05] <wikibugs>	 Analytics / General/Unknown: Raw webrequest partitions for 2014-08-23T20:xx:xx not marked successful - https://bugzilla.wikimedia.org/69971#c1 (christian) a:christian Neither ulsfo, nor eqiad hosts are affected Only esams hosts are affected.  Some hosts are having missing duplicates (like amssq57.es...
[13:48:29] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Document some filter idiosyncrasies through tests [analytics/webstatscollector] - https://gerrit.wikimedia.org/r/156050 (owner: QChris)
[13:50:32] <YuviPanda>	 milimetric: not sure if I've shown you, but http://tools.wmflabs.org/notebooks/yuvipanda/test/test :)
[13:50:40] <YuviPanda>	 works for all toollabs users
[13:51:51] <milimetric>	 that's cool YuviPanda  :)
[13:52:10] <milimetric>	 I've never played with ipython notebooks, maybe I'll do that today
[13:52:12] <YuviPanda>	 milimetric: also have reasonably secure notebooks running on tools as well, accessible from a single script.
[13:52:19] <YuviPanda>	 milimetric: github.com/yuvipanda/labsbooks
[13:52:32] <YuviPanda>	 milimetric: although after talking to halfak, I realized publishing would be an easier problem to tackle first
[13:54:21] <YuviPanda>	 milimetric: re: cache headers + CORS, we could implement that in python or in nginx. I usually prefer the latter, but what do you think?
[13:55:26] <milimetric>	 YuviPanda: we already enabled CORS in apache but it's just not puppetized
[13:55:39] <YuviPanda>	 milimetric: I also emailed analytics list about anonymizing 'edits per country' (and perhaps 'editors per country') and having it available on labsdb, and then wikimetrics can use that as well
[13:55:45] <YuviPanda>	 milimetric: ah, cool, I can puppetize that then :)
[13:55:48] <milimetric>	 it seems easiest in that apache config in puppet I linked you to the other day
[13:55:51] <YuviPanda>	 milimetric: can you give me access?
[13:55:56] <YuviPanda>	 to the project on labs
[13:56:17] <milimetric>	 wikimetrics is in the analytics project, i'll check you're allowed
[13:56:54] <YuviPanda>	 oh, ok
[13:57:56] <milimetric>	 YuviPanda: I added you as an admin, you were already on.  You can use wikimetrics-dev1.eqiad.wmflabs to test, feel free to do absolutely anything you want there, except make sure to protect the local un-pushed commits in the puppet repo.  Those are setting up the private passwords etc.
[13:58:06] <milimetric>	 in /var/lib/git/operations/puppet
[13:58:11] <YuviPanda>	 milimetric: ah, ok :)
[13:58:22] <YuviPanda>	 milimetric: I can puppetize the debs from your requirements.txt if you want as well
[13:58:57] <milimetric>	 YuviPanda: nah, that's ok for now
[13:59:07] <milimetric>	 wikimetrics is intentionally built to stay in labs
[13:59:14] <YuviPanda>	 right
[14:00:02] <milimetric>	 there are a few people who want it in prod but I think if it was there, there'd be less pressure on us to make datasets public
[14:00:15] <milimetric>	 (which is something I want - mooar public datas!) :)
[14:19:51] <YuviPanda>	 milimetric: :D indeed
[14:25:12] <grrrit-wm1>	 (PS2) Ottomata: Show failure reason, if test fails [analytics/webstatscollector] - https://gerrit.wikimedia.org/r/156051 (owner: QChris)
[14:25:17] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Show failure reason, if test fails [analytics/webstatscollector] - https://gerrit.wikimedia.org/r/156051 (owner: QChris)
[16:10:49] <grrrit-wm1>	 (Abandoned) MarkTraceur: Add preference counts to the generated stats [analytics/multimedia] - https://gerrit.wikimedia.org/r/145595 (owner: MarkTraceur)
[16:34:26] <grrrit-wm1>	 (PS2) Milimetric: Set up layout and basic pieces [analytics/dashiki] - https://gerrit.wikimedia.org/r/155826
[16:59:30] <grrrit-wm1>	 (PS2) BearND: Add Apps tab with the first app specific graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155830
[16:59:32] <grrrit-wm1>	 (PS1) BearND: Update gitignore for Mac and IntelliJ files [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/156144
[17:01:06] <grrrit-wm1>	 (CR) Yuvipanda: [C: 2] Update gitignore for Mac and IntelliJ files [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/156144 (owner: BearND)
[17:01:14] <grrrit-wm1>	 (Merged) jenkins-bot: Update gitignore for Mac and IntelliJ files [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/156144 (owner: BearND)
[17:15:24] <grrrit-wm1>	 (PS3) BearND: Add Apps tab with the first app specific graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155830
[17:36:01] <grrrit-wm1>	 (PS4) BearND: Add Apps tab with the first app specific graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155830
[17:36:40] <bearND>	 milimetric: ^ only updated the commit message to mention the new dashboard/apps
[17:36:53] <bearND>	 the main changes are in PS3
[17:45:35] <grrrit-wm1>	 (CR) Milimetric: [C: 2] Add Apps tab with the first app specific graph [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155830 (owner: BearND)
[17:46:12] <bearND>	 milimetric:: Thank you! \o/
[17:46:32] <milimetric>	 bearND: http://mobile-reportcard.wmflabs.org/dashboards/apps
[17:47:17] <milimetric>	 getting your changes up there requires merging them and running a deployer.  The deployer is fairly simple but you need ssh access so just ping me if you want to deploy for now
[17:47:41] <milimetric>	 I don't mind doing it once a week or so, if it's more often I'll get you access
[17:55:48] <bearND>	 milimetric: I can check locally, now that the first report is up, I feel more confident about creating new graphs. I don't mind waiting with a new deploy until I've got all the others finished. So, hopefully only one or two deploys but probably this week.
[17:56:08] <milimetric>	 k bearND
[18:04:34] <yuvipanda>	 milimetric: bearND I suppose I can still deploy as well, assuming it's the same fab script
[18:04:57] <milimetric>	 yuvipanda: it is, just "fab mobile_reportcard deploy.only_data"
[18:05:06] <yuvipanda>	 milimetric: cool :)
[18:05:13] <yuvipanda>	 bearND: so I can deploy too, to give milimetric  a break :)
[18:05:15] <milimetric>	 bearND: https://github.com/wikimedia/limn-deploy that's the deployer
[18:08:17] <bearND>	 milimetric: thanks
[19:48:39] <grrrit-wm1>	 (PS7) QChris: Stop scheduling new recurrent runs if databases lag [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507)
[19:50:50] <grrrit-wm1>	 (CR) Milimetric: [C: 2] Stop scheduling new recurrent runs if databases lag [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507) (owner: QChris)
[19:51:46] <grrrit-wm1>	 (CR) QChris: Stop scheduling new recurrent runs if databases lag (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507) (owner: QChris)
[19:53:40] <milimetric>	 thanks qchris!
[19:53:53] <qchris>	 milimetric: thanks for the catch and for the quick merge.
[19:54:07] <qchris>	 I'll prepare the change to the puppet module then.
[19:54:29] <milimetric>	 thanks very much
[20:09:43] <Ironholds>	 kraken peeps, we have a pull request on github. Not sure if it's duplicated to gerrit, so thought I would let you know:)
[20:13:12] <ottomata>	 hm, welp, kraken is mostly deprecated, and we aren't using that script anymore
[20:13:22] <ottomata>	 i recently mentioned this fact in the Camus thread about it
[20:13:36] <ottomata>	 its still a useful script, probably someone was clued into it and started using it
[20:19:28] * Ironholds nods
[20:41:31] <ottomata>	 qchris:
[20:41:37] <ottomata>	 ha
[20:41:38] <ottomata>	 uh
[20:41:39] <ottomata>	 check this out
[20:41:40] <ottomata>	 https://cwiki.apache.org/confluence/display/Hive/HCatalog+DynamicPartitions#HCatalogDynamicPartitions-ExternalTables
[20:41:54] * qchris looks
[20:41:54] <ottomata>	 Ironholds: thanks, I am reading about this because of that pull request
[20:42:36] <ottomata>	 i'm testing it now, hcat.dynamic.partitioning.custom.pattern does let it see newly created directories...but I don't see it auto-creating any partitions yet
[20:43:00] <ottomata>	 ah, it says 0.13.0 and later
[20:43:03] <ottomata>	 we are still on .12.0
[20:44:09] <qchris>	 Looking forward to Hive 0.13.0.
[20:44:19] <qchris>	 Yup. Hive 0.12.0-cdh5.0.2.
[20:44:30] <ottomata>	 haha, right!  all that work we did!
[20:44:42] <ottomata>	 well, ha, i suppose its all still good
[20:44:54] <ottomata>	 a lot of it is around the sequence stats calculation soooooo
[20:44:55] <ottomata>	 not for nothing!
[20:45:10] <qchris>	 Yes. Agreed.
[20:45:19] <ottomata>	 but still, with our version as is, if I create a new directory with our layout
[20:45:19] <ottomata>	 and then run
[20:45:29] <ottomata>	 MSCK REPAIR TABLE
[20:45:30] <ottomata>	 i get
[20:45:30] <ottomata>	 Partitions not in metastore:	webrequest_dynamic:webrequest_mobile/hourly/2014/08/25/01
[20:47:55] <qchris>	 I do not understand ... Gonna check the table.
[20:50:55] <qchris>	 ottomata: How would your MSCK command know about your location layout (like the "/hourly/" in the name)
[20:51:21] <ottomata>	 qchris
[20:51:22] <ottomata>	 set  hcat.dynamic.partitioning.custom.pattern='webrequest_${webrequest_source}/hourly/${year}/${month}/${day}/${hour}';
[20:51:46] <ottomata>	 all I did was
[20:51:51] <qchris>	 Oh. you set that :-) Cool.
[20:51:56] <ottomata>	 ja, all i did was
[20:52:03] <ottomata>	 1. create table with LOCATION '/user/otto/data/external/webrequest_dynamic'
[20:52:13] <ottomata>	 2. mkdir hdfs dfs -mkdir -p data/external/webrequest_dynamic/webrequest_mobile/hourly/2014/08/25/01;
[20:52:22] <ottomata>	 3. set  hcat.dynamic.partitioning.custom.pattern='webrequest_${webrequest_source}/hourly/${year}/${month}/${day}/${hour}';
[20:52:29] <ottomata>	 4. MSCK REPAIR TABLE webrequest_dynamic;
[20:52:34] <ottomata>	 BAM
[20:52:34] <ottomata>	 Partitions not in metastore:	webrequest_dynamic:webrequest_mobile/hourly/2014/08/25/01
[20:52:34] <qchris>	 (I get the same error message without the 'SET hcat...')
[20:53:59] <qchris>	 I guess we have to wait for Hive 0.13.0 :-)
[20:55:11] <ottomata>	 yeah, and then what, just change the oozie job to run MSCK REPAIR TABLE, instead of ALTER TABLE ADD PARTITOIN?  iunno
[20:55:14] <ottomata>	 what we have works...
[20:55:14] <ottomata>	 :)
[20:55:21] <ottomata>	 either way, cool.
[20:55:25] <qchris>	 Totally!
[20:56:53] <ottomata>	 so, qchris, are you working right now? kinda late for you
[20:57:15] <qchris>	 Yup, still working.
[20:57:31] <qchris>	 Probably like at least for another hour.
[20:57:43] <qchris>	 Gotta finish the wikimetrics stuff.
[20:57:43] <ottomata>	 so, what's up with webstatscollector then?  i see you got it working somehow, whatdya do?
[20:57:50] <ottomata>	 ok
[20:58:15] <qchris>	 The webstatscollector stuff is easier to talk about than to write.
[20:58:18] <qchris>	 Trap?
[20:59:51] <qchris>	 ottomata: ^
[20:59:57] <ottomata>	 ooook headphones...
[21:03:32] <ottomata>	 qchris hallooo
[21:12:42] <tnegrin>	 hey
[21:13:25] <kevinator>	 hey
[22:48:47] <grrrit-wm1>	 (PS1) BearND: Remove end of time span in new app-edits-start report [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/156204
[22:48:49] <grrrit-wm1>	 (PS1) BearND: +app edit reports for previews, saveAttempts, successful edits [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/156205