[02:56:20] <wikibugs>	 Analytics / EventLogging: Multiple user_ids per username in account creation events from ServerSideAccountCreation log - https://bugzilla.wikimedia.org/66101#c15 (Matthew Flaschen) (In reply to Chris Steipp from comment #14) > If we prevent the local account from being created at this point, then any >...
[11:31:36] <grrrit-wm1>	 (PS4) QChris: Stop scheduling new recurrent runs if databases lag [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507)
[11:31:38] <grrrit-wm1>	 (PS4) QChris: When testing, stub out replication lag checking [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155004
[11:44:59] <grrrit-wm1>	 (CR) QChris: Stop scheduling new recurrent runs if databases lag (11 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507) (owner: QChris)
[12:00:54] <grrrit-wm1>	 (CR) Nuria: [WIP] Initial Layout for dashiki (7 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/154787 (owner: Nuria)
[12:01:13] <grrrit-wm1>	 (PS2) Nuria: Setup packages and basic html layout [analytics/dashiki] - https://gerrit.wikimedia.org/r/154787
[12:03:48] <grrrit-wm1>	 (CR) QChris: [C: 2 V: 2] "Would not know how to test, but setting the property for" [analytics/refinery] - https://gerrit.wikimedia.org/r/155439 (owner: Ottomata)
[12:33:05] <grrrit-wm1>	 (CR) QChris: [C: -1] Add --location CLI opt to refinery-drop-webrequest-partitions (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/155281 (owner: Ottomata)
[13:07:36] <wikibugs>	 Analytics / General/Unknown: geowiki data aggregation failed on 2014-08-19 - https://bugzilla.wikimedia.org/69812#c2 (christian) NEW>RESO/FIX Todays run again passed without issues.  Discussing the issue with springle, there was a backup job a bit before the geowiki run. That run spiked a few gra...
[13:41:28] <ottomata>	  ha, qchris_away, i don't understand your 'vertical alignments and diffs :-D' comment on that change!
[13:41:46] <qchris>	 Ignore it.
[13:41:51] <ottomata>	 ha ok
[13:41:58] <qchris>	 I just do not like aligned assignments.
[13:42:03] <ottomata>	 Oh
[13:42:04] <qchris>	 It messes up diffs.
[13:42:19] <ottomata>	 pssh, we look at the code more than the diffs
[13:42:21] <ottomata>	 i value it more :)
[13:42:32] <qchris>	 :-D
[13:42:47] <grrrit-wm1>	 (PS3) Ottomata: Add --location CLI opt to refinery-drop-webrequest-partitions [analytics/refinery] - https://gerrit.wikimedia.org/r/155281
[13:43:06] <ottomata>	 so, the umask thing works with camus, i tried it
[13:43:08] <ottomata>	 but
[13:43:22] <ottomata>	 for an unknown reason, camus isnot creating the files with proper group ownership
[13:43:36] <ottomata>	 and I can't deploy the umask change until the group ownership is right
[13:43:46] <ottomata>	 its creating the files as group owned by 'hadoop'
[13:43:57] <ottomata>	 even though the parent directory is group owned by 'analytics-privatedata-users'
[13:44:37] <qchris>	 Should file group ownership be inherited from the parent directory on Hadoop? That sounds strange.
[13:44:51] <ottomata>	 yes
[13:44:54] <ottomata>	 BSD style :p
[13:45:05] <qchris>	 :-D
[13:45:10] <ottomata>	 When a file or directory is created, its owner is the user identity of the client process, and its group is the group of the parent directory (the BSD rule).
[13:45:12] <ottomata>	 http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html
[13:45:19] <ottomata>	 i did manual tests to be sure of this
[13:45:41] <ottomata>	 the new directories that camus creates have proper groups
[13:45:46] <ottomata>	 its just hte files that don't
[13:46:00] <ottomata>	 OHHHH
[13:46:04] <ottomata>	 i know!
[13:46:06] <ottomata>	 yay for talking!
[13:46:10] <ottomata>	 camus creates the files in a temp dir
[13:46:11] <ottomata>	 pretty sure
[13:46:28] <qchris>	 Ok.
[13:47:00] <ottomata>	 ahhhhh
[13:47:00] <ottomata>	 yes yes
[13:47:02] <ottomata>	 ok
[13:47:06] <ottomata>	 i just have to chgrp the temp dir
[13:47:07] <ottomata>	 i betcha!
[13:47:09] <qchris>	 :-D
[13:50:38] <grrrit-wm1>	 (CR) QChris: [C: -1] Add --location CLI opt to refinery-drop-webrequest-partitions (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/155281 (owner: Ottomata)
[13:52:16] <ottomata>	 PSHHH doh
[13:52:18] <grrrit-wm1>	 (PS4) Ottomata: Add --location CLI opt to refinery-drop-webrequest-partitions [analytics/refinery] - https://gerrit.wikimedia.org/r/155281
[13:53:53] <grrrit-wm1>	 (CR) QChris: [C: 2 V: 2] Add --location CLI opt to refinery-drop-webrequest-partitions [analytics/refinery] - https://gerrit.wikimedia.org/r/155281 (owner: Ottomata)
[13:57:06] <ottomata>	 chgrp on the temp dirs did it!
[13:58:39] <qchris>	 \o/
[13:59:07] <qchris>	 So we revert the umask change?
[14:02:16] <grrrit-wm1>	 (PS1) QChris: Revert "Set HDFS umask-mode=037 when Camus webrequest import runs" [analytics/refinery] - https://gerrit.wikimedia.org/r/155540
[14:02:49] <ottomata>	 nono, we need both
[14:03:00] <ottomata>	 umask does mode, chgrp does group ownership
[14:04:27] <grrrit-wm1>	 (Abandoned) QChris: Revert "Set HDFS umask-mode=037 when Camus webrequest import runs" [analytics/refinery] - https://gerrit.wikimedia.org/r/155540 (owner: QChris)
[14:04:32] <qchris_meeting>	 k
[14:56:24] <grrrit-wm1>	 (PS1) Yuvipanda: Reduce connection pooling recylcing time [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155551
[14:57:44] <grrrit-wm1>	 (CR) Yuvipanda: [C: 2] Reduce connection pooling recylcing time [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155551 (owner: Yuvipanda)
[14:57:52] <grrrit-wm1>	 (Merged) jenkins-bot: Reduce connection pooling recylcing time [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155551 (owner: Yuvipanda)
[14:58:20] <wikibugs>	 Analytics / Wikimetrics: Story:d WikimetricsUser runs 'Rolling Recurring old active editors' report - https://bugzilla.wikimedia.org/69569 (nuria)
[15:04:23] <grrrit-wm1>	 (PS1) Ottomata: Camus created directories need to be executable too [analytics/refinery] - https://gerrit.wikimedia.org/r/155553
[15:05:06] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] Camus created directories need to be executable too [analytics/refinery] - https://gerrit.wikimedia.org/r/155553 (owner: Ottomata)
[15:18:18] <grrrit-wm1>	 (PS1) Ottomata: 027 is the correct umask [analytics/refinery] - https://gerrit.wikimedia.org/r/155562
[15:18:29] <grrrit-wm1>	 (CR) Ottomata: [C: 2 V: 2] 027 is the correct umask [analytics/refinery] - https://gerrit.wikimedia.org/r/155562 (owner: Ottomata)
[15:25:56] <ottomata>	 phew, finally, got it!
[15:26:00] <ottomata>	 groups and permissions all set up right
[15:26:03] <ottomata>	 alright!
[15:52:22] <wikibugs>	 Analytics / Refinery: Raw webrequest partition monitoring did not flag data for 2014-08-18T13:..:.. as valid for text caches - https://bugzilla.wikimedia.org/69854 (christian) NEW p:Unprio s:normal a:None The imported raw webrequests data from text caches for 2014-08-18T13:..:.. at    hdfs:...
[15:54:51] <wikibugs>	 Analytics / Refinery: Raw webrequest partition monitoring did not flag data for 2014-08-18T13:..:.. as valid for text caches - https://bugzilla.wikimedia.org/69854#c1 (christian) Created attachment 16256   --> https://bugzilla.wikimedia.org/attachment.cgi?id=16256&action=edit kafka-requests-per-second-...
[15:55:06] <wikibugs>	 Analytics / Refinery: Raw webrequest partition monitoring did not flag data for 2014-08-18T13:..:.. as valid for text caches - https://bugzilla.wikimedia.org/69854#c2 (christian) NEW>RESO/WON Monitoring worked as expected, as the data is missing sequence numbers:    +-----------------------------...
[16:03:14] <grrrit-wm1>	 (Abandoned) Yuvipanda: Remove extra comma [analytics/quarry/web] - https://gerrit.wikimedia.org/r/154008 (owner: Helder.wiki)
[16:27:36] <wikibugs>	 Analytics / Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://bugzilla.wikimedia.org/69615#c2 (christian) Just to help us keep track how relevant this is, it again happened on    2014-08-17T20:xx:xx/2014-08-17T21:xx:xx (on upload)   2014-08-18T20:xx...
[16:30:16] <grrrit-wm1>	 (Abandoned) Yuvipanda: [WIP] Add DataTables [analytics/quarry/web] - https://gerrit.wikimedia.org/r/151006 (owner: Yuvipanda)
[16:30:21] <wikibugs>	 Analytics / Quarry: Support queries containing non-latin strings - https://bugzilla.wikimedia.org/69786#c1 (Yuvi Panda) NEW>RESO/FIX Fixed now! http://quarry.wmflabs.org/query/295 is same query and returns Resultset (1569 rows)
[16:30:24] <grrrit-wm1>	 (PS1) Yuvipanda: Set db to utf8 [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155576
[16:31:57] <grrrit-wm1>	 (CR) Yuvipanda: [C: 2] Set db to utf8 [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155576 (owner: Yuvipanda)
[16:32:02] <grrrit-wm1>	 (Merged) jenkins-bot: Set db to utf8 [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155576 (owner: Yuvipanda)
[16:37:30] <grrrit-wm1>	 (PS1) QChris: DO NOT MERGE HiveQL file to check for sequence stats across hours [analytics/refinery] - https://gerrit.wikimedia.org/r/155578 (https://bugzilla.wikimedia.org/69615)
[16:38:20] <grrrit-wm1>	 (Abandoned) QChris: DO NOT MERGE HiveQL file to check for sequence stats across hours [analytics/refinery] - https://gerrit.wikimedia.org/r/155578 (https://bugzilla.wikimedia.org/69615) (owner: QChris)
[16:39:33] <YuviPanda>	 milimetric: nuria hey! cache setup for your public files sounds fun :)
[16:40:02] <YuviPanda>	 since I've to do that in Quarry as well
[16:40:15] <YuviPanda>	 milimetric: nuria I think proper way is to cache them on the webserver as well :)
[16:41:05] <wikibugs>	 Analytics / Refinery: Raw webrequest partition monitoring did not flag data for 2014-08-18T13:..:.. as valid for text caches - https://bugzilla.wikimedia.org/69854#c3 (Andrew Otto) Ha, ah yes, ok, if this corresponds with an election, then this makes sense.  The producers themselves have errors in the...
[16:45:42] <nuria>	 \o/ YuviPanda
[16:47:28] <nuria>	 YuviPanda are we talking about this: https://bugzilla.wikimedia.org/show_bug.cgi?id=68445?
[16:49:21] <YuviPanda>	 nuria: yeah
[16:49:22] <YuviPanda>	 I am
[16:50:00] <nuria>	 k, then...what do you mean with caching them on webserver?
[16:50:33] <nuria>	 YuviPanda: cause they are "static" files
[16:52:12] <YuviPanda>	 nuria: oh, right. then that's not needed. Just CORS and cache headers, then.
[16:53:15] <nuria>	 ah, ok, YuviPanda
[17:04:06] <wikibugs>	 Analytics / Visualization: Story: EEVSUser loads static site in accordance to Pau's design - https://bugzilla.wikimedia.org/67806 (Dan Andreescu)
[17:04:36] <wikibugs>	 Analytics / Wikimetrics: replication lag may affect recurrent reports - https://bugzilla.wikimedia.org/68507 (Dan Andreescu)
[17:06:21] <wikibugs>	 Analytics / Wikimetrics: Story: EEVS user does not see reports for projects without databases - https://bugzilla.wikimedia.org/69297 (Dan Andreescu)
[17:09:52] <wikibugs>	 Analytics / Visualization: Story: AnalyticsEng has website for EEVS - https://bugzilla.wikimedia.org/68351 (Dan Andreescu)
[17:14:06] <milimetric>	 YuviPanda: https://git.wikimedia.org/tree/operations%2Fpuppet%2Fwikimetrics
[17:30:09] <grrrit-wm1>	 (CR) Milimetric: "Most of the minor stuff is ok, except a small change on the query." (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507) (owner: QChris)
[17:35:03] <grrrit-wm1>	 (PS4) Milimetric: Add an option to schedule reports for all cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/150440
[17:37:00] <grrrit-wm1>	 (PS3) Milimetric: Ignore private or invalid wikis [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/154864
[17:38:30] <grrrit-wm1>	 (CR) Nuria: [C: 2] Add an option to schedule reports for all cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/150440 (owner: Milimetric)
[17:38:37] <grrrit-wm1>	 (Merged) jenkins-bot: Add an option to schedule reports for all cohorts [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/150440 (owner: Milimetric)
[17:42:52] <grrrit-wm1>	 (CR) Nuria: [C: 2] Ignore private or invalid wikis [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/154864 (owner: Milimetric)
[17:42:59] <grrrit-wm1>	 (Merged) jenkins-bot: Ignore private or invalid wikis [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/154864 (owner: Milimetric)
[17:43:29] <qchris>	 milimetric: about the factory. Is it ok if I just remove it?
[17:43:33] <qchris>	 We had two rounds, and I could not convince you that they are worth it.
[17:43:37] <qchris>	 So it's best to get rid of it.
[17:43:40] <qchris>	 I do not think that nu ria also needs to waste time on it.
[17:43:59] <milimetric>	 ok qchris, that sounds fine
[17:44:05] <milimetric>	 I like the idea but I'd like to stay consistent
[17:44:06] <wikibugs>	 Analytics / Wikimetrics: Story: EEVS user does not see reports for projects without databases - https://bugzilla.wikimedia.org/69297#c2 (nuria) I missed the bug Id on commit message.   Here is the patch that fixes this issue:  https://gerrit.wikimedia.org/r/#/c/154864/
[17:44:08] <qchris>	 k.
[17:44:20] <wikibugs>	 Analytics / Wikimetrics: Story: EEVS user does not see reports for projects without databases - https://bugzilla.wikimedia.org/69297 (nuria) NEW>RESO/FIX
[17:44:40] <qchris>	 nuria: Since mili metric asked you to review the factory in https://gerrit.wikimedia.org/r/#/c/155003
[17:44:45] <qchris>	 no need to spend time on it.
[17:44:50] <qchris>	 I'll remove the factory.
[17:44:52] <nuria>	 ok qchris
[17:45:00] <nuria>	 very well
[17:46:29] <nuria>	 FYI qchris we are going to deploy to wikimetrics prod
[17:46:39] <qchris>	 Great!
[17:47:25] <grrrit-wm1>	 (Abandoned) QChris: When testing, stub out replication lag checking [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155004 (owner: QChris)
[17:50:08] <ottomata1>	 qchris: got a sec to talk about fairscheduler queues w me?
[17:50:26] <qchris>	 Sure.
[17:50:37] <ottomata>	 so, this is what we have now
[17:50:37] <ottomata>	 https://github.com/wikimedia/operations-puppet/blob/production/templates/hadoop/fair-scheduler.xml.erb
[17:50:44] <ottomata>	 and, it doesn't actually make that much sense
[17:51:05] <ottomata>	 people's hive queries are going in the default queue...by default
[17:51:12] <qchris>	 Yup.
[17:51:40] <ottomata>	 we only really  need 2 queues, right?
[17:51:43] <ottomata>	 default and...admin?
[17:51:51] <ottomata>	 where admin has higher weight
[17:52:03] <qchris>	 For a start, it seems that would do. Yes.
[17:52:11] <qchris>	 But admin sounds ... funny :-)
[17:52:37] <qchris>	 But let's stick to admin, so we have a name for the discussion.
[17:53:01] <ottomata>	 yeah, name coudl be whatever
[17:53:04] <ottomata>	 don't care
[17:53:07] <ottomata>	 haha, well i probably do care
[17:53:34] <ottomata>	 ha, so, yeah, its hard to say what we really need, all we know is that jobs in default queue shouldn't ever keep jobs in admin queue from running
[17:53:59] <qchris>	 Is the fair scheduler the right one then?
[17:54:11] <qchris>	 (I haven't looked at them in detail)
[17:55:46] <ottomata>	 i think so, there are a few ways to prioritize queues
[17:55:50] <ottomata>	 weight being one
[17:55:56] <ottomata>	 the other is specific min resource requirements
[17:56:02] <ottomata>	 but those are in RAM, cores
[17:56:38] <qchris>	 k
[17:57:40] <ottomata>	 ha, ok, yeah, hard to say, so.  i think we just need default and admin
[17:57:49] <ottomata>	 and for now, will do admin weighted 2
[17:58:23] <qchris>	 2 queues sounds good.
[17:58:34] <qchris>	 Making admin more important sounds fine too.
[17:59:05] <ottomata>	 ok, name preference?  not admin?
[18:00:07] <qchris>	 The queue is not for admin tasks ... so admin does not sound too nice.
[18:00:15] <ottomata>	 priority?
[18:00:38] <qchris>	 That sounds better ... but will it confuse people if the queue is called like a property?
[18:01:22] <qchris>	 "high-priority" would sound less confusing to me.
[18:01:35] <qchris>	 but I do not like that too much either.
[18:01:46] <qchris>	 urgent
[18:01:51] <qchris>	 important
[18:01:53] <qchris>	 crucial
[18:02:22] <qchris>	 mhmmm...
[18:02:42] <qchris>	 automatic
[18:02:49] <qchris>	 refinery
[18:02:52] <ottomata>	 hahsa
[18:03:46] <qchris>	 immediate, important
[18:04:17] <ottomata>	 essential
[18:04:25] <qchris>	 essential is great!
[18:04:53] <ottomata>	 hmm, ok
[18:04:54] <qchris>	 preemptive
[18:05:18] <qchris>	 prerogative
[18:05:51] <qchris>	 fundamental
[18:05:53] <qchris>	 leading
[18:06:04] <qchris>	 indispensable
[18:06:35] <qchris>	 backbone
[18:07:39] <ottomata>	 enterprise
[18:07:40] <ottomata>	 haha
[18:07:44] <qchris>	 :-P
[18:08:05] <ottomata>	 let's do essential, hm, so, i don't really feel like restarting the oozie jobs for this............right now anyway
[18:08:12] <ottomata>	 should I just wait til we have more changes?
[18:08:29] <qchris>	 I think it's ok to wait.
[18:08:33] <ottomata>	 like, maybe our idea to fire up a secondary check for the sequence number/hour boundry mixup?
[18:08:57] <qchris>	 Sounds good to defer until then.
[18:09:14] <ottomata>	 ok
[19:24:08] <grrrit-wm1>	 (PS1) Yuvipanda: Add user groups and the sudo functionality [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155606
[19:24:14] <grrrit-wm1>	 (CR) jenkins-bot: [V: -1] Add user groups and the sudo functionality [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155606 (owner: Yuvipanda)
[19:25:38] <grrrit-wm1>	 (PS2) Yuvipanda: Add user groups and the sudo functionality [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155606
[19:26:10] <grrrit-wm1>	 (CR) Yuvipanda: [C: 2] Add user groups and the sudo functionality [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155606 (owner: Yuvipanda)
[19:26:15] <grrrit-wm1>	 (Merged) jenkins-bot: Add user groups and the sudo functionality [analytics/quarry/web] - https://gerrit.wikimedia.org/r/155606 (owner: Yuvipanda)
[21:13:03] <grrrit-wm1>	 (PS1) BearND: Update README [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155636
[21:13:05] <grrrit-wm1>	 (PS1) BearND: Add SQL for Apps - Impressions on edit pencil (per day) [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155637
[21:45:26] <grrrit-wm1>	 (CR) Yuvipanda: [C: 2] Update README [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155636 (owner: BearND)
[21:45:31] <grrrit-wm1>	 (Merged) jenkins-bot: Update README [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/155636 (owner: BearND)
[21:54:54] <grrrit-wm1>	 (PS5) QChris: Stop scheduling new recurrent runs if databases lag [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507)
[21:54:56] <grrrit-wm1>	 (PS1) QChris: Drop unused imports [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155644
[21:58:59] <ottomata>	 qchris: still around?
[21:59:03] <qchris>	 Yup.
[21:59:12] <qchris>	 For 57 seconds. :-)
[21:59:15] <ottomata>	 got a sec to help me make sense of something, just wanna talk it out
[21:59:15] <qchris>	 48.
[21:59:16] <ottomata>	 ah!
[21:59:21] <ottomata>	 i'm in the trap!~
[21:59:22] <qchris>	 Sure.
[21:59:26] <qchris>	 Joining ...
[22:53:19] <grrrit-wm1>	 (CR) QChris: Stop scheduling new recurrent runs if databases lag (3 comments) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/155003 (https://bugzilla.wikimedia.org/68507) (owner: QChris)