[00:34:25] <wikibugs>	 10Analytics, 06Commons, 06Multimedia, 10Tabular-Data, and 3 others: Allow structured datasets on a central repository (CSV, TSV, JSON, GeoJSON, XML, ...) - https://phabricator.wikimedia.org/T120452#2859028 (10Yurik) 05Open>03Resolved Enabled in https://gerrit.wikimedia.org/r/#/c/326046/ - T148745  I th...
[01:11:52] <wikibugs_>	 06Analytics-Kanban, 06Wikipedia-iOS-App-Backlog, 10iOS-app-feature-Analytics: Drop in iOS app pageviews since version 5.2.0 - https://phabricator.wikimedia.org/T148663#2859061 (10Tbayer) Confirming that the app's registered pageviews appear to have gone back up to normal on or around November 10 (the above c...
[09:36:08] <wikibugs_>	 06Analytics-Kanban: Webrequest dataloss registered during the last kafka restarts - https://phabricator.wikimedia.org/T152674#2859440 (10JAllemandou) @ottomata: Looks like we've not been able to repro just by restarting ... Maybe  some weird race-condition of some sort ?
[10:11:45] <wikibugs>	 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 07Puppet: Refactor eventlogging.pp role into multiple files (and maybe get rid of inheritance) - https://phabricator.wikimedia.org/T152621#2855084 (10elukey) Last action left: removing unnecessary hiera data belonging to the eventlogging role (that is r...
[10:12:40] <elukey>	 a-team: I applied some patches for https://phabricator.wikimedia.org/T152621
[10:13:10] <elukey>	 so removing the old eventlogging role from eventlog1001 (deleted) and re-added your ssh keys because they got removed
[10:13:16] <elukey>	 so it was a no-op
[10:13:33] <elukey>	 and EL looks fine
[10:13:34] <elukey>	 BUT
[10:13:46] <elukey>	 please double check and follow up in here if anything is wrong
[10:15:56] * elukey afk!
[11:03:03] <grrrit-wm>	 (03PS6) 10Joal: Add mediawiki history spark jobs to refinery-job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) 
[12:21:00] <grrrit-wm>	 (03PS1) 10Addshore: In quantities units of "1" are converted to "" [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/326104 (https://phabricator.wikimedia.org/T152615) 
[12:22:08] <grrrit-wm>	 (03PS1) 10Addshore: In quantities units of "1" are converted to "" [analytics/wmde/toolkit-analyzer-build] - 10https://gerrit.wikimedia.org/r/326105 (https://phabricator.wikimedia.org/T152615) 
[12:23:11] <grrrit-wm>	 (03CR) 10Addshore: [V: 032 C: 032] In quantities units of "1" are converted to "" [analytics/wmde/toolkit-analyzer] - 10https://gerrit.wikimedia.org/r/326104 (https://phabricator.wikimedia.org/T152615) (owner: 10Addshore) 
[12:23:24] <grrrit-wm>	 (03CR) 10Addshore: [V: 032 C: 032] In quantities units of "1" are converted to "" [analytics/wmde/toolkit-analyzer-build] - 10https://gerrit.wikimedia.org/r/326105 (https://phabricator.wikimedia.org/T152615) (owner: 10Addshore) 
[12:31:17] <jynus>	 hi there, anyone here to help me later with the analytics db restarts?
[12:31:39] <joal>	 Hi jynus 
[12:31:47] <jynus>	 hello
[12:32:15] <jynus>	 can you make it in around 1 hour?
[12:32:40] <jynus>	 or any other time today?
[12:32:48] <joal>	 jynus: yessir !
[12:33:26] <jynus>	 thank you, I will send an email announcing that
[12:33:26] <joal>	 you let me know when you need me (it means stopping EventLogging processes for the time you work
[12:33:33] <joal>	 Thanks !
[12:33:43] <jynus>	 yes, that will be only for the db1046 (master) restart
[12:33:48] <joal>	 k
[12:33:55] <joal>	 I'll be around
[12:34:00] <jynus>	 I am going to have luch, see you in one hour
[13:41:21] <jynus>	 I am doing the prerequisites for the restarts
[13:52:29] <jynus>	 joal, we can stop eventlogging insertions on the database now
[13:52:45] <joal>	 jynus: ok, give me a few minutes
[13:53:02] <jynus>	 sure
[13:57:46] <joal>	 !log Stopped EventLogging Mysql consumer for database restart
[13:57:46] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:58:50] <joal>	 jynus: insertion process stopped, waiting to have confirmation on EL grafana dashboard
[13:58:51] <jynus>	 ok, I do not see any client
[13:59:07] <jynus>	 when you confirm, I will set all servers in read only mode
[13:59:16] <jynus>	 to avoid accidental writes
[13:59:23] <joal>	 Sounds wise :)
[13:59:43] <icinga-wm>	 PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: consumer/mysql-m4-master-00
[14:00:16] <jynus>	 I will ack that
[14:01:15] <icinga-wm>	 ACKNOWLEDGEMENT - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: consumer/mysql-m4-master-00 Jcrespo stopped for mysql restart
[14:02:08] <joal>	 I was also trying to do it, but I think I don't have rights
[14:02:33] <jynus>	 is everything ok, can I put things in read only?
[14:02:49] <joal>	 jynus: charts are too slow -- I confirm ,everything looks good on my siode
[14:03:02] <joal>	 jynus: go, go, GO :)
[14:03:10] <jynus>	 it is better that I force the read only and things fail
[14:03:19] <joal>	 it is
[14:03:21] <jynus>	 than that they start writing to the wrong server
[14:24:00] <jynus>	 joal, it is coming back soon
[14:24:30] <joal>	 jynus: no problemo :)
[14:25:44] <ashgrigas>	 mforns milimetric nuria please add comments to this in dropbox if there are any outstanding metrics or breakdowns missing today
[14:25:45] <ashgrigas>	 https://www.dropbox.com/s/iv5agq1kp09tnq1/Wikistats%20Nav%20Model%20v2.png?dl=0
[14:26:05] <ashgrigas>	 updated based on yesterday's chat and including metrics from the future reports wiki dan mentioned
[14:26:28] <jynus>	 this specific server model takes a lot to reboot
[14:26:48] <mforns>	 ashgrigas, thanks will look into it
[14:27:18] <joal>	 ashgrigas: do you mind if I have quick look (I promise not to make remarks ;)
[14:31:08] <jynus>	 it is up, confirming everthing is ok
[14:32:00] <jynus>	 joal, let's restart eventlogging writes
[14:32:09] <joal>	 ok jynus, on it
[14:32:43] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning.
[14:32:48] <joal>	 !log restarted eventlogging mysql consumer after DB restart
[14:32:49] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:34:00] <jynus>	 eventlog:  Inserted about 2000 rows
[14:34:06] <jynus>	 seems ok to me
[14:34:24] <jynus>	 do you know a table that is frequently inserted, to check latest timestamps?
[14:34:28] <joal>	 jynus: trying to check logs, but can't find them :)
[14:34:28] <jynus>	 (schema)
[14:34:44] <jynus>	 a direct check will be more reliable
[14:35:30] <jynus>	 Yes, I see the edits coming in https://tendril.wikimedia.org/report/slow_queries?host=^db1046&user=&schema=&qmode=eq&query=&hours=3
[14:35:30] <joal>	 jynus: navigation_timing should be a good one
[14:35:38] <joal>	 ok
[14:36:09] <jynus>	 things may be a bit slower for a while
[14:36:13] <jynus>	 until the memory is hot
[14:36:18] <joal>	 jynus: sorry not to be able to pop fast answers, EL is not the system I know best :)
[14:37:39] <joal>	 jynus: data flows in graphite, everyhting seems to be working as expected
[14:38:15] <jynus>	 good
[14:38:22] <jynus>	 I will now restart the slaves
[14:38:29] <jynus>	 these are more delicate ones
[14:38:29] <joal>	 Thanks jynus for this update :)
[14:38:42] <jynus>	 but more impacting to normal users
[14:38:51] <ashgrigas>	 joal feel free to comment as well
[14:38:58] <ashgrigas>	 open to any feedback :) 
[14:39:00] <joal>	 many thanks ashgrigas :)
[15:09:25] <jynus>	 things should be back to normal
[15:09:36] <jynus>	 regarding analycs db restarts
[15:09:43] <jynus>	 I will do some sanity checks
[15:09:50] <jynus>	 and send an email annoucing that
[16:01:04] <nuria>	 joal: standduppp?
[16:01:18] <joal>	 soooorry
[17:25:25] <mforns>	 milimetric, joal, do you want to wikistats?
[17:25:40] <milimetric>	 in da cave
[17:25:48] <mforns>	 ok
[17:25:58] <joal>	 mforns: in 1-1, will be there in 5 minutes
[17:26:04] <mforns>	 ok joal 
[17:26:50] <milimetric>	 nuria: cc ^
[17:26:55] <milimetric>	 oh!
[17:26:56] <milimetric>	 sorry
[17:29:30] <nuria>	 milimetric: can you come to batcave now?
[17:29:39] <milimetric>	 we're in the cave!
[17:29:46] <milimetric>	 maybe there's a parallel universe again
[17:29:52] <milimetric>	 stupid shroedinger
[17:30:32] <nuria>	 milimetric: coming
[17:58:45] <wikibugs_>	 06Analytics-Kanban: POC of clickhouse + denormalize edit data on labs that can be used as a backend for quarry - https://phabricator.wikimedia.org/T152788#2860153 (10Nuria)
[17:59:34] <wikibugs>	 10Analytics: POC of clickhouse + denormalize edit data on labs that can be used as a backend for quarry - https://phabricator.wikimedia.org/T152788#2860165 (10Nuria)
[18:26:51] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 10EventBus, 13Patch-For-Review: Ensure no dropped messages in eventlogging producers when stopping broker - https://phabricator.wikimedia.org/T142430#2534697 (10Volans) Please see T152771, we had to revert this last change to unbreak Puppet on `stat1002`.
[18:48:21] <grrrit-wm>	 (03PS1) 10Nuria: Fixing tests on master [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/326152 (https://phabricator.wikimedia.org/T152631) 
[18:49:26] <grrrit-wm>	 (03PS2) 10Nuria: Fixing tests on master [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/326152 (https://phabricator.wikimedia.org/T152631) 
[18:50:39] <nuria>	 mforns, milimetric : changed deps a bit to fix tests please remember to npm install sinon and jasmine-core again when testing patch. Rolled jasmine-core forward and sinon back a bit
[18:51:04] <grrrit-wm>	 (03PS7) 10Joal: Add mediawiki history spark jobs to refinery-job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) 
[18:51:40] <joal>	 milimetric, mforns : Just pushed PS7 - I'll keep working on docs, I'm not finished, but will not review ;)
[18:52:17] <joal>	 I think from a code perspective, toplevel MediawikiHistoryRunner and whole denormalized package are ok to review
[18:52:27] <joal>	 milimetric, mforns --^
[18:58:07] <joal>	 With that, I take my leave for this week :)
[18:58:21] <joal>	 Have a good weekend a-team !
[18:58:49] <milimetric>	 have a good one!
[20:30:47] <grrrit-wm>	 (03PS1) 10EBernhardson: Lucene Stemmer UDF [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/326168 (https://phabricator.wikimedia.org/T148811) 
[20:33:41] <ashgrigas>	 @milimetric @mforns can you give me some examples of period breakdown contents
[20:34:06] <ashgrigas>	 is it days, months, year etc
[20:34:14] <ashgrigas>	 what increments if so
[20:34:39] <milimetric>	 ashgrigas: yes, days, months, years, sometimes hours or quarters maybe
[20:35:01] <milimetric>	 we were thinking those would be part of a more generic set of controls that would apply to any visualizations
[20:35:13] <ashgrigas>	 specific groupings? like past 30 days past year etc
[20:35:25] <milimetric>	 it would allow users to toggle between available breakdowns in the same way across reports
[20:35:49] <ashgrigas>	 yeah just thinking about layout
[20:35:54] <ashgrigas>	 so wanted to know what those will be
[20:35:58] <ashgrigas>	 the common set
[20:36:03] <milimetric>	 right
[20:36:18] <milimetric>	 mmm, it's hard to have a common set
[20:36:29] <milimetric>	 because some reports only make sense at the monthly or yearly level and daily doesn't make sense
[20:36:32] <milimetric>	 or vice versa
[20:36:42] <mforns>	 ashgrigas, I think there are 2 concepts in time handling: time range and time breakdown
[20:36:42] <ashgrigas>	 i mean the generic set of controls above for breakdowns
[20:36:58] <ashgrigas>	 you mention
[20:37:00] <mforns>	 time range is a from - to time span
[20:37:12] <ashgrigas>	 right thats a custom range right
[20:37:17] <ashgrigas>	 user could select start/end
[20:37:31] <ashgrigas>	 versus preset time breakdowns like past year past month?
[20:37:35] <milimetric>	 right, but "last 30 days" could also be a range
[20:37:47] <mforns>	 and breakdown is granularity, can be: total, yearly, quarterly, monthly, weekly, daily, hourly...
[20:37:50] <milimetric>	 no, breakdowns would be more like the granularity shown
[20:37:51] <milimetric>	 right
[20:38:15] <mforns>	 but I find presets OK as well, depending on the case
[20:38:33] <milimetric>	 pivot does a good job of this, what do you think mforns ?
[20:38:34] <ashgrigas>	 how much granularity do we want to support
[20:38:50] <ashgrigas>	 preset ranges, preset summaries/totals for a time period
[20:38:53] <milimetric>	 what I've seen so far is daily, monthly, yearly, and rarely hourly and weekly
[20:38:56] <ashgrigas>	 and custom ranges and custom summaries?
[20:38:58] <mforns>	 milimetric, yes pivot is nice, but too technical I think
[20:39:00] <mforns>	 for wikistats
[20:39:07] <milimetric>	 I mean just the date selection
[20:39:20] <milimetric>	 that's too technical too?
[20:39:26] <ashgrigas>	 ok can we go with Day/Month/Year for now
[20:39:59] <mforns>	 milimetric, xD, I mean that in pivot you have to drag the time dimension as a breakdown and then open it and select a granularity... I think that is too technical
[20:40:01] <ashgrigas>	 both range and summary
[20:40:29] <milimetric>	 mforns: no, I meant just the time range/granularity selection, not the dimension / metric / split stuff
[20:40:35] <mforns>	 ashgrigas, what do you mean with summary?
[20:40:55] <ashgrigas>	 totals
[20:40:58] <ashgrigas>	 versus ranges
[20:41:10] <ashgrigas>	 thought those were the two concepts dan was talking about
[20:41:36] <milimetric>	 no, that's a different thing
[20:41:48] <milimetric>	 if we're talking breakdown vs. range, we mean like this:
[20:42:00] <milimetric>	 if I choose breakdown: days, range: "last week"
[20:42:03] <mforns>	 milimetric, I see... mmmm for my taste it's still too technical, but it's just me
[20:42:05] <milimetric>	 then I'd get 7 points
[20:42:08] <ashgrigas>	 i see
[20:42:09] <ashgrigas>	 ok
[20:42:12] <ashgrigas>	 got it
[20:42:15] <milimetric>	 or breakdown: months, range: "last year"
[20:42:17] <milimetric>	 I'd get 12
[20:42:31] <ashgrigas>	 can't we say time frame as the starting selection
[20:42:41] <ashgrigas>	 and then within that choose the granularity that makes most sense
[20:42:47] <ashgrigas>	 for that graph
[20:42:52] <ashgrigas>	 instead of having it a user selection?
[20:43:04] <milimetric>	 well, sure, we'll have good defaults
[20:43:08] <mforns>	 ashgrigas, makes sense
[20:43:17] <milimetric>	 but in some cases people do want to choose
[20:43:25] <ashgrigas>	 i think people may think first about the time period they want answers to
[20:43:29] <mforns>	 sure, there should be a selector with a default
[20:43:30] <ashgrigas>	 and then second the granularity
[20:43:33] <ashgrigas>	 but just an assumption
[20:43:38] <milimetric>	 that's why in wikistats there are tables that show things by month, year, etc.
[20:43:48] <milimetric>	 I agree, first range, then granularity
[20:43:48] <mforns>	 ashgrigas, I am with you in the assumption :]
[20:43:55] <milimetric>	 and usually the default granularity would be good
[20:43:59] <mforns>	 yea
[20:44:06] <ashgrigas>	 right
[20:44:06] <milimetric>	 so the ability to customize it could be hidden and more "advanced"
[20:44:17] <ashgrigas>	 so granularity = breakdown in our spreadsheet
[20:44:32] <mforns>	 also, some metrics won't have all granularity options, some even may have just one
[20:44:55] <milimetric>	 breakdown is a more general term, but in the case of time it applies to granularity, yes
[20:45:00] <mforns>	 ashgrigas, yes, granularity = days means breakdown by days
[20:45:15] <mforns>	 xD duplicate answers
[20:46:00] <ashgrigas>	 ok granularity is easier for me to get
[20:46:02] <ashgrigas>	 somehow
[20:49:35] <mforns>	 ashgrigas, you could have different granularities in another dimension, for example: "size of the articles"
[20:49:40] <mforns>	 0-1000 words | 1000 - 100000 words | more than 100000 words (bigger granularity)
[20:49:47] <mforns>	 or 0-100 words | 100 - 1000 words | 1000-10000 words | 10000 - 100000 words | more than 100000 words (thinner granularity)
[20:50:44] <ashgrigas>	 sure
[20:51:00] <ashgrigas>	 you could also handle granularity in the visualization itself if a small enough set of options
[20:51:03] <milimetric>	 you just mean the word "granularity" better describes it than "breakdown"?  I agree with that
[20:51:10] <ashgrigas>	 yeah
[20:51:44] <milimetric>	 yep, that works for me: we can use granularity and range
[20:52:10] <mforns>	 I like the word granularity for time as well
[20:52:37] <ashgrigas>	 of the metrics we have - how many have multiple dimensions of granularity ie. period and project
[20:52:39] <milimetric>	 it's ok for other dimensions too, really, it comes from size of sand grains, so applies equally well to anything
[20:52:58] <ashgrigas>	 or language and content type
[20:53:03] <milimetric>	 any of the dimensions can technically have a granularity associated
[20:53:18] <ashgrigas>	 you mean metric
[20:53:18] <milimetric>	 like if you look at project + language together, granularities could be:
[20:53:20] <ashgrigas>	 metrics
[20:53:30] <milimetric>	 "project", "language", "both"
[20:54:00] <milimetric>	 maybe this will help:
[20:54:05] <ashgrigas>	 i guess im asking if we support use selecting multiple breakdowns/granularities for a given metric
[20:54:07] <ebernhardson>	 is there a pre-existing mapping table in hive for translating the names in webrequest.pageview_info['project'] into mediawiki db names? (ex: en.wikipedia -> enwiki)
[20:54:15] <ebernhardson>	 i can create one, but if something already exists no need
[20:54:24] <ashgrigas>	 or if they can just see one breakdown/granularity at a time
[20:54:40] <ashgrigas>	 (user)
[20:55:03] <milimetric>	 I haven't thought about that yet, ashgrigas, good question
[20:55:11] <milimetric>	 either way is possible
[20:55:26] <milimetric>	 ebernhardson: there will be one soon, I have one in my user db, one sec
[20:55:30] <ashgrigas>	 it could be quite complex if they can start selecting multiple breakdowns on a given metric visualization
[20:55:50] <milimetric>	 yep, it can get crazy quickly
[20:56:03] <milimetric>	 they're not called hypercubes for nothing :)
[20:56:42] <milimetric>	 ebernhardson: the table is called milimetric.mediawiki_namespace_map and the columns are hostname and dbname, but they're duplicated for each namespace so select distinct
[20:56:57] <milimetric>	 ebernhardson: and this will soon be wmf.mediawiki_namespace_map, like over the next week or so
[20:57:01] <ashgrigas>	 well we should decide soon :)
[20:57:08] <ashgrigas>	 since it will impact the ui elements
[20:57:11] <ebernhardson>	 milimetric: perfect, thanks!
[20:57:22] <milimetric>	 ashgrigas: I saw this a bit further in the design though, way after the navigation
[20:57:46] <ashgrigas>	 i think these things inform the navigation
[20:57:59] <ashgrigas>	 because we need to get a sense of how many controls there may be per topic
[20:58:07] <ashgrigas>	 to know how people should move between topics etc
[20:58:32] <milimetric>	 ok, I see
[20:58:49] <mforns>	 ashgrigas, we should decide between more options + single visualization, versus less options + multiple visualizations
[20:59:03] <milimetric>	 but this definitely needs more thought then, we have to pin down very specifically what we need for each metric
[20:59:19] <milimetric>	 so far I only made sure to organize the high levels (metrics and topics)
[20:59:50] <milimetric>	 these are the next levels: (dimensions)
[21:00:04] <mforns>	 yes agree
[21:00:21] <ashgrigas>	 i thought that was the breakdowns category in my IA map
[21:00:29] <ashgrigas>	 is dimensions something different?
[21:01:15] <mforns>	 ashgrigas, a dimension is an attribute of your data set that lets you break it down
[21:01:33] <ashgrigas>	 i think we have the option to have a main visualization with controls for metrics and breakdowns for each topic (3 total) on the dashboard
[21:01:45] <mforns>	 for example, if your data set is about articles, a dimension can be the language
[21:01:49] <ashgrigas>	 or we can have one master visualization that has more chrome and controls to move between topics
[21:02:03] <mforns>	 all articles have the language attribute and you can use it to break the data down
[21:02:22] <ashgrigas>	 mforns ok so that seems in sync with the breakdowns column of the map right?
[21:02:44] <ashgrigas>	 do those things need to be more defined and mapped directly to each of our metrics then
[21:02:49] <ashgrigas>	 in a way they're aren't currently
[21:02:52] <mforns>	 yes, dimensions and breakdowns belong to each other
[21:03:31] <mforns>	 I guess it would be good to have a more detailed study of breakdowns/dimensions per metric
[21:04:08] <mforns>	 in my experience, though, having a visualization with more than 2 breakdowns is not really good...
[21:04:17] <mforns>	 counting time in that
[21:04:23] <ashgrigas>	 agree
[21:04:33] <ashgrigas>	 i think time (range) and one dimension is enough
[21:04:36] <ashgrigas>	 to digest
[21:04:50] <milimetric>	 ebernhardson: I just made wmf.mediawiki_namespace_map if you need to depend on it in a job, and when we productionize it it'll just update more regularly
[21:04:50] <mforns>	 well, time range is not a breakdown :]
[21:05:02] <mforns>	 time granularity is the breakdown
[21:05:11] <ashgrigas>	 right sorry
[21:05:22] <ebernhardson>	 milimetric: job probably wont be merged till next week or the following anyway so no rush, but thanks!
[21:05:42] <mforns>	 ok ok, you can have a visualization that is a pie chart, for example, and it will have a time range, but no time breakdown
[21:05:57] <ebernhardson>	 i still have a problem with this hive query where i can't figure out how to process the webrequest table 1 hour at a time, and doing a whole day used 1TB of memory so probably not an option :)
[21:06:05] <milimetric>	 hm... ashgrigas if that's the case, we may need to think of other navigation elements
[21:06:12] <ebernhardson>	 (the problem is i need the timespan + 1 hour, for sessionizing)
[21:06:13] <milimetric>	 ebernhardson: feel free to gist us
[21:06:29] <mforns>	 ashgrigas, like this sunburst chart: https://edit-analysis.wmflabs.org/compare/
[21:06:49] <mforns>	 it has a time range selector, but it has no time breakdown
[21:07:00] <milimetric>	 ashgrigas / mforns: if we don't want more than one breakdown per chart, we still have the problem of lots of charts being very similar except for a dimension
[21:07:18] <mforns>	 milimetric, I think 2 breakdowns is possible (counting time)
[21:07:19] <milimetric>	 so we're just factoring that out to more metrics or more variations of metrics
[21:07:34] <milimetric>	 yeah, but some of these graphs / tables have more than 2 breakdowns
[21:08:03] <ashgrigas>	 can you give an example of having more than 2 that makes the data story clearer
[21:08:10] <ashgrigas>	 and is still digestible
[21:08:47] <mforns>	 milimetric, yes, but the data is kind of crammed, no?
[21:08:54] <milimetric>	 ok, but sorry if this confuses the "granularity/range" ideas :)
[21:09:01] <milimetric>	 ok, so one example is active editors
[21:09:05] <mforns>	 aha
[21:09:18] <milimetric>	 the way this is broken down currently is bot vs. anonymous vs. registered users
[21:09:26] <milimetric>	 so that's one way
[21:09:31] <mforns>	 ok
[21:09:43] <milimetric>	 also, we have numbers for "active" meaning 1 per month, 5 per month, 100 per month
[21:09:54] <milimetric>	 we could call those "new", "active", and "very active"
[21:10:00] <milimetric>	 but we could also include them together
[21:10:01] <milimetric>	 so that's two
[21:10:23] <milimetric>	 then there's how many of those there are per month and per year
[21:10:30] <milimetric>	 so that's three
[21:11:01] <milimetric>	 and I guess there's how many there are in a specific project or deduplicated across all projects
[21:11:05] <mforns>	 milimetric, yes the dimensions are there, but I don't think you could visualize editors per month, per user type and per activity at the same time
[21:11:23] <milimetric>	 why not?
[21:11:24] <mforns>	 the chart would be a mess
[21:11:29] <milimetric>	 bots making at least 5 edits per month
[21:11:29] <mforns>	 unless you have a table
[21:11:35] <milimetric>	 anon making at least 100 edits per month
[21:11:37] <mforns>	 oh!
[21:11:44] <milimetric>	 not all at the same time
[21:11:45] <mforns>	 but that is not a breakdown, that is a filter!
[21:11:52] <milimetric>	 no it's not :)
[21:12:16] <mforns>	 how not, you're selecting an option in each dimension, no?
[21:12:50] <mforns>	 if you would visualize that in pivot, you'd use the filter component
[21:12:53] <milimetric>	 right, but you have 3 dimensions including time
[21:13:14] <milimetric>	 so you're talking about showing it at the same time, what pivot calls "split", right?
[21:13:18] <mforns>	 sure! 3 dimensions are ok, but not 3 breakdowns at the same time
[21:13:33] <mforns>	 N filters are ok I think
[21:13:52] <mforns>	 but I'd say no more than 2 breakdowns including time
[21:14:03] <milimetric>	 honestly this is too confusing to talk about in IRC without looking at a UI
[21:14:07] <mforns>	 xD
[21:14:08] <milimetric>	 very easy to misunderstand
[21:14:23] <mforns>	 we can batcave maybe?
[21:14:31] <milimetric>	 I agree, I'm not sure we're making sense for ashgrigas 
[21:17:21] <mforns>	 yea, split = breakdown in my head
[21:23:19] <ashgrigas>	 ok i have about 15 mins
[21:23:22] <ashgrigas>	 want to chat briefly
[21:23:34] <mforns>	 ok ashgrigas, milimetric?
[21:23:57] <milimetric>	 omw
[21:24:33] <mforns>	 ashgrigas, https://hangouts.google.com/hangouts/_/wikimedia.org/a-batcave
[21:37:45] <ebernhardson>	 milimetric: this basically explains my problem with trying to select a given hour, and the next hour: https://phabricator.wikimedia.org/T152804
[21:42:40] <ebernhardson>	 sigh, i made a task instead of a paste :P sec
[21:43:22] <ebernhardson>	 https://phabricator.wikimedia.org/P4604
[21:47:06] <ebernhardson>	 it needed because basically what i'm doing is joining search logs against webrequest, if a search happens at 23:59:59 i need to find the clicks which happen at 00:05:12 the next day
[21:48:44] <milimetric>	 ebernhardson: do you know about LEAD?
[21:49:03] <ebernhardson>	 milimetric: i'm using that for sessionizing yes, but it didn't seem to work for handling which partitions to query
[21:49:19] <mforns>	 bye team, have a nice weekend :]
[21:49:23] <milimetric>	 nite mforns 
[21:49:28] <mforns>	 :]
[21:50:02] <milimetric>	 ebernhardson: wait... but if you're joining you know exactly what partition... I'm still confused
[21:50:06] * milimetric reads again
[21:50:19] <ebernhardson>	 milimetric: i need two partitions, i need the current hour and the next hour
[21:50:32] <ebernhardson>	 i can write something that works for selecting hours 22 and 23, but i cant seem to select hour 23 and hour 00 from next day
[21:51:04] <milimetric>	 doesn't just date arithmetic work?
[21:51:13] <milimetric>	 like "add 1 hour"?
[21:51:29] <milimetric>	 (that's convoluted as hell in hive, but possible)
[21:51:29] <ebernhardson>	 milimetric: hive doesn't have an add 1 hour :( what it does have is from_unixtime(to_unixtime(timestamp)+3600)
[21:51:35] <milimetric>	 right
[21:51:40] <ebernhardson>	 but that's what i'm using in the pastie, which selects all partitions instead of 2 partitions
[21:51:58] <ashgrigas>	 milimetric mforns https://docs.google.com/spreadsheets/d/1M5m6z7YWSqxB3rqQstxPVBK-J5s70p1XMrTN7gYetC8/edit?usp=sharing
[21:52:00] <ashgrigas>	 updated tab
[21:52:11] <milimetric>	 ashgrigas: cool, thx
[21:53:09] <milimetric>	 (ebernhardson I'm gonna try executing this)
[21:53:24] <ebernhardson>	 milimetric: should be safe, the explain dependency means it doesn't actually query anything just reports what would happen
[21:54:10] <milimetric>	 I'm trying just the date arithmetic, curious why it doesn't work
[21:57:55] <milimetric>	 ebernhardson: I think row_date needs ${year} instead of plain "year" and the same for the others
[21:58:01] <milimetric>	 (in its SET declaration)
[21:58:16] <ebernhardson>	 milimetric: wouldn't work, it needs to be the date of the row, not the provided date to the script
[21:58:20] <milimetric>	 oddly that seems to execute fine but throw an error when I try to select it explicitly
[21:58:24] <milimetric>	 oh I see
[21:58:26] <milimetric>	 it's dynamic
[21:58:27] <milimetric>	 duh
[21:58:38] <milimetric>	 'cause hive works on replacement, right
[21:58:51] <ebernhardson>	 yea, it's literally a string replacement, no parsing is done on the hivevar's
[22:00:26] <ebernhardson>	 hmm, maybe i can get away with some IF(${hour}=23, ..., ...) that just does the right thing in each case
[22:02:00] <milimetric>	 ebernhardson: just noticed it's not zero-padding the month and day, that probably messes with the string compare
[22:02:07] <milimetric>	 but you do TO_DATE
[22:02:07] <milimetric>	 right
[22:02:31] <milimetric>	 well, no, wait, these are strings
[22:02:50] <milimetric>	 the to_date is just to combine them, yeah, it might just be 'cause of the zero-padding
[22:03:14] <milimetric>	 2016-11-4 10:00:00 is greater than 2016-11-12 00:00:00
[22:04:11] <ebernhardson>	 oh, hmm that might be it
[22:04:16] <milimetric>	 ebernhardson: I did "select ${row_timestamp} from wmf.webrequest where year=2016 and month=11 and day=4 and hour=10 limit 1;"
[22:04:33] <milimetric>	 (the limit 1 was very important, heh :))
[22:04:35] <ebernhardson>	 :)
[22:04:43] <ebernhardson>	 i have a meeting, but i'll be back to figuring this out in an hour
[22:04:50] <milimetric>	 i've gotta run, good luck!
[22:04:55] <milimetric>	 have a nice weekend all
[22:06:25] <ebernhardson>	 milimetric: enjoy!