[00:07:51] <wikibugs>	 Analytics, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707539 (GWicke)
[00:08:10] <wikibugs>	 Analytics, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707320 (GWicke)
[00:08:57] <wikibugs>	 Analytics, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707320 (GWicke) I went ahead and hijacked the task description with a summary of my own. Please edit to reflect the discussion more accurately!
[00:09:21] <wikibugs>	 Analytics, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707542 (GWicke)
[00:11:13] <wikibugs>	 Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Microtask: Create a very simple REST API - https://phabricator.wikimedia.org/T114838#1707546 (jgbarah) NEW
[00:12:11] <wikibugs>	 Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Microtask: Create a very simple REST API - https://phabricator.wikimedia.org/T114838#1707546 (jgbarah)
[00:14:52] <wikibugs>	 Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1707561 (jgbarah)
[00:16:40] <wikibugs>	 Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1088193 (jgbarah) >>! In T60585#1706581, @Oakshweta11 wrote: > @jgbarah could you suggest what I should do next?   I would...
[00:18:15] <wikibugs>	 Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1707575 (jgbarah) >>! In T60585#1677452, @Simmimourya3107 wrote: > Is there any chance that this project gets proposed for...
[01:49:01] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707671 (Milimetric)
[01:54:16] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1707673 (Milimetric) Here's what I understood from our last discussion, I think it's fairly close to the task summary, but it's a bit simpler in my mind.  We have two ways of fro...
[02:47:08] <wikibugs>	 Analytics-Tech-community-metrics: Microtask: Create a very simple REST API - https://phabricator.wikimedia.org/T114838#1707725 (NiharikaKohli)
[07:24:48] <wikibugs>	 Analytics-Tech-community-metrics: Port MediaWikiAnalysis to SQLAlchemy - https://phabricator.wikimedia.org/T114437#1707946 (Anmolkalia) Hi, @jgbarah, I am facing a problem in running the original mediawiki_analysis.py. I am writing this in the terminal "./mediawiki_analysis.py --database mwdb --db-user root...
[07:44:54] <wikibugs>	 Analytics-Tech-community-metrics: Port MediaWikiAnalysis to SQLAlchemy - https://phabricator.wikimedia.org/T114437#1707960 (jgbarah) >>! In T114437#1704569, @Anmolkalia wrote: > Hi, I have a doubt. The dbms we will be using in the backend is till MySQL, right? So, I should continue using MySQL datatypes? Sib...
[07:47:12] <wikibugs>	 Analytics-Tech-community-metrics: Port MediaWikiAnalysis to SQLAlchemy - https://phabricator.wikimedia.org/T114437#1707962 (jgbarah) >>! In T114437#1704738, @Anmolkalia wrote: > Hi, @jgbarah, this is the what the mapping file looks like. Let me know if this is fine. {F2661511}  At first glance, it seems reas...
[07:49:03] <wikibugs>	 Analytics-Tech-community-metrics: Port MediaWikiAnalysis to SQLAlchemy - https://phabricator.wikimedia.org/T114437#1707963 (jgbarah) >>! In T114437#1707946, @Anmolkalia wrote: > Hi, @jgbarah, I am facing a problem in running the original mediawiki_analysis.py. I am writing this in the terminal "./mediawiki_a...
[11:48:47] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production. - https://phabricator.wikimedia.org/T114881#1708445 (mforns) NEW a:mforns
[11:50:12] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production {dove} - https://phabricator.wikimedia.org/T114881#1708457 (mforns)
[11:58:55] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production {dove} - https://phabricator.wikimedia.org/T114881#1708469 (mforns)
[12:03:43] <krrrit-wm>	 (PS2) Joal: Add CassandraXSVLoader to refinery-job [analytics/refinery/source] - https://gerrit.wikimedia.org/r/232448 (https://phabricator.wikimedia.org/T108174)
[12:18:48] <wikibugs>	 Analytics-Backlog, Analytics-Wikimetrics: Some special characters break Wikimetrics' encoding {dove} - https://phabricator.wikimedia.org/T114884#1708492 (mforns) NEW
[12:19:06] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production {dove} - https://phabricator.wikimedia.org/T114881#1708501 (mforns) The problem was that a user inserted a tag with the character(s): ``` ՛ր ``` And it seems that the encoding we use does not support that. It...
[12:19:40] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production {dove} [3 pts] - https://phabricator.wikimedia.org/T114881#1708504 (mforns)
[12:33:52] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production {dove} [2 pts] - https://phabricator.wikimedia.org/T114881#1708524 (mforns)
[12:40:44] <wikibugs>	 Analytics-Backlog, Analytics-Wikimetrics: Some special characters break Wikimetrics' encoding {dove} - https://phabricator.wikimedia.org/T114884#1708539 (mforns)
[12:42:59] <krrrit-wm>	 (PS4) Joal: [WIP] Add cassandra load job for pageview API [analytics/refinery] - https://gerrit.wikimedia.org/r/236224 (https://phabricator.wikimedia.org/T108174)
[12:45:41] <krrrit-wm>	 (CR) Joal: [C: -1] "Need to remove comments on pom.xml" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/232448 (https://phabricator.wikimedia.org/T108174) (owner: Joal)
[13:33:23] <wikibugs>	 Analytics-Tech-community-metrics, DevRel-October-2015: Tech community KPIs for the WMF metrics meeting - https://phabricator.wikimedia.org/T107562#1708649 (Qgil)
[13:33:38] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-October-2015: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#1708654 (Qgil)
[14:06:43] <wikibugs>	 Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on terbium - https://phabricator.wikimedia.org/T112660#1708738 (Ottomata) See:  https://phabricator.wikimedia.org/T109567#1683710 and https://phabricator.wikimedia.org/T114199
[14:07:07] <wikibugs>	 Analytics-Backlog, Analytics-EventLogging: Upgrade eventlogging servers to Jessie - https://phabricator.wikimedia.org/T114199#1708740 (Ottomata)
[14:07:09] <wikibugs>	 Analytics-EventLogging, Performance-Team, Patch-For-Review: Support kafka in eventlogging client on terbium - https://phabricator.wikimedia.org/T112660#1708739 (Ottomata)
[14:17:57] <wikibugs>	 Analytics, Services: restbase is not listening on port 7231 on aqs* - https://phabricator.wikimedia.org/T114742#1708783 (mobrovac) Open>Resolved AQS is now up && running, resolving.
[14:17:59] <wikibugs>	 Analytics-Kanban, netops, operations, Patch-For-Review: Puppetize a server with a role that sets up Cassandra on Analytics machines [13 pts] {slug} - https://phabricator.wikimedia.org/T107056#1708785 (mobrovac)
[14:25:57] <wikibugs>	 Analytics-Tech-community-metrics, Possible-Tech-Projects, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1708828 (Oakshweta11) I'll start with T114838: Microtask: Create a very simple REST API Thanks.
[14:27:17] <nuria>	 ottomata: i did miss a lot, eh? what? you are writing js now?
[14:28:04] <ottomata>	 haha
[14:28:06] <ottomata>	 yes! :)
[14:30:40] <krrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Setting up wmf branch to use archiva.wikimedia.org and maven release plugin [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/244043 (owner: Ottomata)
[14:31:36] <ottomata>	 hehe, ooo
[14:31:39] <ottomata>	 it does look similar
[14:31:43] <ottomata>	 oops, wrong chat
[14:35:09] <nuria>	 ottomata: do you have a sec?
[14:35:50] <krrrit-wm>	 (PS1) Ottomata: Fix scm url for gerrit camus [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/244161
[14:36:07] <krrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Fix scm url for gerrit camus [analytics/camus] (wmf) - https://gerrit.wikimedia.org/r/244161 (owner: Ottomata)
[14:36:57] <ottomata>	 nuria:  yes
[14:37:12] <nuria>	 ottomata: okquestion about madhuvishy 's patch
[14:37:31] <nuria>	 ottomata: seems to me that we do not want schemas here: https://gerrit.wikimedia.org/r/#/c/243990/
[14:37:49] <nuria>	 ottomata: but rather in an outside depot that both mediawiki and our java apps can share
[14:38:23] <nuria>	 ottomata: because we will not be able to deploy the
[14:38:32] <nuria>	 ottomata: refinery camus to mw stack, right?
[14:38:42] <ottomata>	 nuria:  eventually yes
[14:38:44] <ottomata>	 but we don't ahve that yet
[14:38:51] <ottomata>	 and we can't create it now without a lot of bikeshedding
[14:39:04] <nuria>	 ottomata: but search code is  almost done right?
[14:39:42] <nuria>	 ottomata: they are having to duplicate those schemas to unit test at least
[14:39:46] <nuria>	 ottomata: as things are now
[14:40:14] <nuria>	 ottomata: right?
[14:40:24] <nuria>	 ottomata: so we are having schemas n two places
[14:40:26] <ottomata>	 nuria:  at the moment, yes, these schemas will be duplicated.
[14:40:36] <ottomata>	 we will talk about this problem on thursday :)
[14:40:47] <nuria>	 ottomata: ok
[14:52:48] <nuria>	 joal: i own you a million crs
[14:53:04] <nuria>	 ottomata: did you put the camus artifacts in archiva?
[14:55:10] <krrrit-wm>	 (CR) Nuria: "Let's talk about this today." (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 (owner: Madhuvishy)
[14:55:18] <wikibugs>	 Analytics-Kanban: Spike: understand wikistats enough to estimate replacing pageview data source {lama} [8 pts] - https://phabricator.wikimedia.org/T114660#1708905 (Milimetric) a:Milimetric
[14:56:00] <ottomata>	 nuria:  am trying...
[14:56:13] <ottomata>	 we are trying to make the camus wmf branch able to mvn release to archiva
[14:56:35] <nuria>	 ottomata: every time it builds?
[14:56:39] <ottomata>	 no
[14:56:42] <nuria>	 ottomata: ahhh
[14:56:44] <ottomata>	 the same way we do for refinery
[14:56:45] <wikibugs>	 Analytics-Kanban: Gain permission to delete articles on wikitech (needed for doc cleanup) [3 pts] - https://phabricator.wikimedia.org/T114672#1708908 (Milimetric) a:Milimetric
[14:57:12] <wikibugs>	 Analytics-Kanban: Gain permission to delete articles on wikitech (needed for doc cleanup) [3 pts] - https://phabricator.wikimedia.org/T114672#1708914 (Nuria) And mediawiki, right?
[15:00:04] <joal>	 nuria: don't worry :)
[15:00:18] <nuria>	 joal: which one is the most pressing one?
[15:00:42] <nuria>	 ottomata: let me know if i can help with archiva stuff, did madhu "faked" maven yesterday to build her jars?
[15:00:45] <nuria>	 cc madhuvishy
[15:00:50] <milimetric>	 joal: keyspaces are there!
[15:01:22] <joal>	 nuria: https://gerrit.wikimedia.org/r/#/c/240868/ would be great (not in detail, camus needs to be pushed to archiva first, but the approach and so
[15:01:29] <joal>	 milimetric: awesome !
[15:02:26] <joal>	 ottomata: Thanks for the camus mvn stuff !
[15:04:02] <wikibugs>	 Analytics, Privacy, Varnish: Connect Hadoop records of the same request coming via different channels - https://phabricator.wikimedia.org/T113817#1708950 (Nuria) > but we should not make correlating the webrequest/pageviews dataset with mediawiki logs easier, mmm... this is precisely what this ticket i...
[15:04:27] <nuria>	 joal: looking
[15:05:23] <nuria>	 joal: idea is to use a spark job for monitoring correct?
[15:05:25] <ottomata>	 joal: nuria, it smostly working, i'm trying to do release:perform now
[15:05:38] <ottomata>	 i think it sworking
[15:05:43] <joal>	 nuria: it's not even spark: it's core java :)
[15:06:01] <joal>	 ottomata: you have 0.1.0-wmf6 in archiva (com.linkedin.*
[15:06:02] <nuria>	 ottomata: so,  should i be able to build
[15:06:28] <joal>	 ottomata: is that what is expected?
[15:06:39] <joal>	 nuria: I think you should be able to build my code yes :)
[15:06:44] <nuria>	 ottomata: Still:Failed to execute goal on project refinery-camus: Could not resolve dependencies for project org.wikimedia.analytics.refinery.camus:refinery-camus:jar:0.0.20-SNAPSHOT: The following artifacts could not be resolved: com.linkedin.camus:camus-schema-registry:jar:0.1.0-wmf6, com.linkedin.camus:camus-etl-kafka:jar:0.1.0-wmf6,
[15:06:44] <nuria>	 com.linkedin.camus:camus-kafka-coders:jar:0.1.0-wmf6: Failure to find com.linkedin.camus:camus-schema-registry:jar:0.1.0-wmf6 in https://archiva.wikimedia.org/repository/mirrored/ was cached in the local repository, resolution will not be reattempted until the update interval of wmf-mirrored has elapsed or updates are forced -> [Help 1]
[15:07:08] <joal>	 mwarf :(
[15:07:50] <joal>	 nuria, that's weird: I can see the jar in archiva
[15:08:06] <nuria>	 joal: man .. mvn everytime!
[15:08:11] <nuria>	 let me re-build
[15:08:12] <joal>	 :D
[15:09:08] <wikibugs>	 Analytics-Kanban: Gain permission to delete articles on wikitech and mediawiki (needed for doc cleanup) [3 pts] - https://phabricator.wikimedia.org/T114672#1708959 (Milimetric)
[15:09:47] <ottomata>	 nuria:  they aren't uploaded yet, am working on it
[15:10:04] <nuria>	 ottomata: k, that's what i thought
[15:10:05] <joal>	 Ah, right, I can see them on search though ... wird
[15:10:13] <madhuvishy>	 Nuria: yes, I did mvn install: install-file locally to get my build to work
[15:10:29] <madhuvishy>	 And yes, those schema registry classes were only for testing
[15:11:24] <madhuvishy>	 The dummy schema one is in a different package now - org.Wikimedia.. And thats the one I'm using in the properties
[15:11:39] <madhuvishy>	 So I'm sure it's using the right class
[15:12:17] <ottomata>	 joal:  i think some of them went...
[15:12:22] <ottomata>	 its in the middle of it
[15:12:25] <ottomata>	 and it failed once before
[15:13:59] <nuria>	 madhuvishy: wait, teh bindings (which is what we use) are in the same package right? https://gerrit.wikimedia.org/r/#/c/243990/3/refinery-camus/src/main/java/com/linkedin/camus/example/records/DummyLog2.java
[15:14:24] <nuria>	 madhuvishy: ah wait , no
[15:14:28] <madhuvishy>	 Yup
[15:14:35] <madhuvishy>	 Its in my new one
[15:15:29] <nuria>	 madhuvishy: right, but on the same "package" com.linkedin.camus.example.records;
[15:15:40] <madhuvishy>	 So, I wanted to test my new setup, so I thought I would make a search schema registry, and make sure everything else is working. When that dint work, I put the dummy one in there so I can see if it was something else
[15:16:03] <nuria>	 madhuvishy: I see, there are several things
[15:16:06] <madhuvishy>	 Nuria: true
[15:16:27] <nuria>	 the dummy one (since it exists on the other jar) if you have the other jar is been loaded from there
[15:16:37] <madhuvishy>	 Can be fixed by changing the namespace on the local class and trying
[15:16:54] <madhuvishy>	 And I dint put the example one on the path
[15:17:05] <nuria>	 ahhh, ok, that makes sense
[15:17:15] <nuria>	 other thing are bindings
[15:17:34] <nuria>	 madhuvishy: does your jar have bindings for cirrus search schema?
[15:17:40] <madhuvishy>	 Yes
[15:17:49] <madhuvishy>	 I pushed it with the source too
[15:19:23] <madhuvishy>	 Its in /home/madhuvishy/avro-kafka/refinery-camus-0.20 something jar
[15:19:26] <joal>	 hey milimetric
[15:19:33] <milimetric>	 hey hey
[15:19:39] <madhuvishy>	 nuria: ^
[15:19:52] <joal>	 the top 'all-days' for the top end-point, is it 'all-day' or 'all-days' ?
[15:20:08] <nuria>	 madhuvishy: ok, and you build it by adding the jars to your ~/.m2?
[15:20:12] <joal>	 milimetric: --^
[15:20:32] <milimetric>	 all-days: https://github.com/wikimedia/restbase/blob/master/specs/analytics/v1/pageviews.yaml#L193
[15:21:33] <milimetric>	 joal: if you're still testing that's cool but I think the per-article data should be loaded first if possible, that's the one people are blocked on us for
[15:22:11] <krrrit-wm>	 (CR) Nuria: [WIP] Add refinery-camus module (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 (owner: Madhuvishy)
[15:22:46] <nuria>	 ay so many things ... i am getting lost ....
[15:23:09] <madhuvishy>	 nuria: yup, let's catch up after standup
[15:23:10] <nuria>	 madhuvishy: ok, i put on the patch all things i wanted to capture
[15:23:39] <nuria>	 madhuvishy: ok
[15:23:51] <madhuvishy>	 I do wanna have a common registry that does topic-schema mapping and returns the appropriate schema
[15:24:01] <madhuvishy>	 Both those registries should ideally be gone
[15:24:03] <nuria>	 madhuvishy: ya, let's do that today
[15:24:18] <madhuvishy>	 But Id first like the search schema to work
[15:24:26] <nuria>	 madhuvishy: teh topic/schema mapping will be on properties file
[15:24:36] <madhuvishy>	 Yup
[15:24:40] <nuria>	 madhuvishy: we will just pass a property that tells us what schema to load
[15:24:46] <nuria>	 and we will load it at runtime
[15:24:49] <joal>	 milimetric: starting tonight (still a few things to have squared0
[15:25:08] <nuria>	 madhuvishy: what did you used to generate bindings?
[15:25:27] <nuria>	 madhuvishy: the mvn plugin?
[15:27:27] <harej>	 In Wikimedia Report Card terms, what does "reach" refer to?
[15:27:38] <harej>	 Further, are there any breakdowns by country, at least for the big countries?
[15:28:00] <madhuvishy>	 nuria: yes, its all in the pom
[15:28:11] <madhuvishy>	 mvn compile will generate it, so will package etc
[15:28:29] <madhuvishy>	 milimetric: ^^
[15:28:42] <nuria>	 madhuvishy: did you build it by putting the jars on your .m2? since they are not in archiva?
[15:28:49] <madhuvishy>	 yes
[15:29:11] <madhuvishy>	 nuria: http://pastebin.com/pue28vUV
[15:30:07] <nuria>	 madhuvishy: ja, so handy! i always did it low tech by hand
[15:30:46] <milimetric>	 harej: it's a comScore term, it means the percentage of people in that region that have visited our projects
[15:31:25] <nuria>	 ottomata: standup baby
[15:31:30] <milimetric>	 harej: I think there are country breakdowns, Erik Zachte usually processes that data and makes it available on a monthly basis
[15:34:11] <ottomata>	 YES [INFO] BUILD SUCCESS for camus.
[15:34:21] <harej>	 So I see for Europe that in June 2015, there were 134.5 million uniques from Europe and the reach was 31.7%. But 31.7% of Europe isn't 134.5 million; that would be closer to the 300-millions. How should these numbers be interpreted?
[15:35:55] <nuria>	 harej: those are devices , not people
[15:36:14] <harej>	 which are devices, not people? reach or uniques?
[15:39:29] <wikibugs>	 Analytics-Kanban: Update camus-wmf to be deployed by maven (missing jars otherwise) {hawk} [8 pts] - https://phabricator.wikimedia.org/T114657#1709089 (Ottomata) a:Ottomata
[15:39:51] <mforns>	 bd808, I implemented the changes you suggested here: https://gerrit.wikimedia.org/r/#/c/241984/ Have you seen them? :]
[15:40:07] <wikibugs>	 Analytics-Kanban: Update camus-wmf to be deployed by maven (missing jars otherwise) {hawk} [8 pts] - https://phabricator.wikimedia.org/T114657#1702293 (Ottomata) Done!  https://archiva.wikimedia.org/#artifact~releases/com.linkedin.camus/camus-etl-kafka/0.1.0-wmf6
[15:40:45] <bd808>	 mforns: I think I saw the email notification but didn't do a full review. I'll do that now and refresh the beta cherry-pick if it looks good
[15:41:17] <mforns>	 bd808, thank you very much!
[15:41:19] <harej>	 milimetric: where can I find the country breakdowns?
[15:41:21] <ottomata>	 bd808:  if you +1 i will merge
[15:41:43] <harej>	 nuria: which refers to devices and not people: reach or unique visitors?
[15:42:04] <milimetric>	 harej: unique visitors refers to devices
[15:42:42] <milimetric>	 harej: reach is basically unique visitors divided by how many devices comScore figures each person accesses our projects from in each different area
[15:43:20] <milimetric>	 to get more data, either email Erik Zachte directly or the analytics list as other people might have crunched the numbers the same way you have
[15:43:46] <harej>	 and our pageview/reach data is down y/y; this makes me sad, unless there is a logical explanation?
[15:43:48] <milimetric>	 harej: this is important though: the comScore data is wildly inaccurate for us because it doesn't include any data from mobile devices.  And it's considered wildly inaccurate anyway
[15:44:01] <milimetric>	 harej: yes, very logical: they don't include mobile devices :)
[15:44:23] <milimetric>	 harej: this is a more accurate count of pageviews: https://vital-signs.wmflabs.org
[15:44:45] <milimetric>	 (there's a decline there but it's mostly seasonal, which will show up once we have more data)
[15:45:10] <milimetric>	 there are annotations at the bottom for some outages and there's a breakdown by type of access on the left, so you can see the mobile data
[15:45:37] <harej>	 and how do I use this site to get data for all the projects?
[15:45:58] <harej>	 I'm not getting any useful information; just blank screens.
[15:46:10] <milimetric>	 harej: search for "totals" on the top left
[15:46:19] <milimetric>	 harej: hm? sounds like an error...
[15:46:30] <milimetric>	 maybe file a bug on phabricator, but try clearing your cache first
[15:46:31] <harej>	 yes; incidentally, how well does this site play with Internet Explorer 11?
[15:46:55] <milimetric>	 oh :) not sure, I just recently got IE 11 so I'll try it now :)
[15:47:57] <nuria>	 harej: we might need to fix some bugs, try chrome /ff
[15:48:36] <harej>	 oh, if I *had* those browsers on this computer I would use them without being prompted to!
[15:48:58] <milimetric>	 harej: just tried on IE 11, works fine.  File a bug with what you're seeing if you're getting blank screens.  My IE 11 loaded everything in about 1 second
[15:49:53] <nuria>	 harej: second milimetric , for your data questions an e-mail to analytics@ would be best
[15:50:17] <harej>	 analytics@lists.wikimedia.org?
[15:50:21] <milimetric>	 yes
[15:50:27] <bd808>	 mforns: I updated the beta cluster cherry-pick. Can you force a few errors so we can see if it still does what we want?
[15:50:39] <mforns>	 bd808, sure, just a sec
[15:54:22] <wikibugs>	 Analytics-Backlog, MediaWiki-API, Reading-Infrastructure-Team, Research-and-Data, Patch-For-Review: Publish detailed Action API request information to Hadoop - https://phabricator.wikimedia.org/T108618#1709138 (bd808)
[15:55:49] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1709142 (GWicke) @milimetric, that's basically option 1). Sounds good to me.  Regarding the root, shall we use `/api/rest_v1/stats/`, `/api/rest_v1/metrics/`, `/api/rest_v1/data/...
[15:56:11] <mforns>	 bd808, error logs are in logstash-beta
[15:57:11] <bd808>	 mforns: https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/eventlogging looks ok to me. Does the data for the events look right to you?
[15:57:15] <wikibugs>	 Analytics-Backlog: Flag in x-analytics in varnish any request that comes with no cookies whatsoever - https://phabricator.wikimedia.org/T114370#1709145 (Nuria) p:Triage>Unbreak!
[15:58:01] <mforns>	 bd808, yes they look good :]
[15:58:10] <bd808>	 sweet.
[15:58:18] <krrrit-wm>	 (Abandoned) Ottomata: [WIP] How to send job results via email using oozie [analytics/refinery] - https://gerrit.wikimedia.org/r/182350 (owner: Nuria)
[15:58:22] <krrrit-wm>	 (Abandoned) Ottomata: Add generic oozie component for emailing data [analytics/refinery] - https://gerrit.wikimedia.org/r/210632 (owner: Madhuvishy)
[16:01:16] <wikibugs>	 Analytics-Backlog: Oozie sends emails when any job fails - https://phabricator.wikimedia.org/T114901#1709169 (JAllemandou) NEW
[16:04:52] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1709189 (Milimetric) Since it's proxying to the analytics query service, we chatted a bit and figured /api/rest_v1/analytics/ would make the most sense.  I agree /data is somethi...
[16:04:59] <ottomata1>	 mforns:  logstash change applied
[16:05:07] <mforns>	 ottomata, mmmmmm
[16:05:36] <bd808>	 prod dashboard at https://logstash.wikimedia.org/#/dashboard/elasticsearch/eventlogging-errors
[16:05:52] <ottomata>	 COOL
[16:08:00] <mforns>	 bd808, no eventlogging error logs are entering production logstash
[16:08:13] <mforns>	 we should have like one event every other second, in average
[16:08:22] <bd808>	 hmmm...
[16:09:01] <bd808>	 I see the kafka input running in the process list
[16:09:05] <mforns>	 aha
[16:09:10] <ebernhardson>	 so, how scalable is this new eventlogging stuff.  I would kinda like to start collecting information about the results people are clicking on from Special:Search (to build a feedback loop) but by definition that feedback loop is only usefull with minimal sampling
[16:10:14] <bd808>	 mforns: the prod connect string looks like this: zk_connect => "conf1001.eqiad.wmnet,conf1002.eqiad.wmnet,conf1003.eqiad.wmnet/kafka/eqiad"
[16:10:17] <ebernhardson>	 our other option is to build out some sort of bounce and suck the data straight out of web request logs, which is also possible
[16:10:26] <bd808>	 is that the right syntax for multiple hosts?
[16:11:04] <mforns>	 bd808, yes it seems right, although the port is not there...
[16:11:28] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1709209 (GWicke) To me, 'metrics' seem to be a bit more accurate description of what is available in that hierarchy. 'Analytics' has a higher-level ring to me. [The wiki says](ht...
[16:11:30] <mforns>	 will ask ot-tomata
[16:12:05] <mforns>	 ebernhardson, we recently did some load testing on EventLogging
[16:12:22] <mforns>	 and achieved more than 7500 evt/sec in a single machine
[16:12:38] <mforns>	 in theory, EventLogging is now lineraly scalable
[16:12:47] <ebernhardson>	 its not writing to a single mysql master anymore either?
[16:13:14] <bd808>	 mforns: the beta config doesn't have a port either: zk_connect => "deployment-zookeeper01.eqiad.wmflabs/kafka/deployment-kafka"
[16:13:14] <mforns>	 so if we add more machines to it, we can continue scaling until...?
[16:13:19] <ebernhardson>	 sweet!
[16:14:08] <mforns>	 ebernhardson, it continues to write to mysql, but in case of such large event flows, we would blacklist the schema
[16:14:22] <mforns>	 and its events would only be stored in hdfs
[16:14:30] <ebernhardson>	 mforns: ok cool, we would only actually want them in hdfs anyways
[16:15:17] <mforns>	 ebernhardson, you should anyway talk to ottomata for the details and confirmation
[16:15:20] <mforns>	 :]
[16:15:49] <ebernhardson>	 sure, and it will probably be a month before we get arround to building out this feedback loop, we have a few other hadoop things in the pipeline first it was just another on our list of ways to improve search :)
[16:15:59] <mforns>	 bd808, you're right, the format seems correct to me
[16:17:54] <ottomata>	 bd808:  that format is correct, it would be good if the ports were there, but i think it picks the default if not
[16:17:55] <ottomata>	 2181
[16:18:01] <ottomata>	 ebernhardson:  everytign mforns says is correct
[16:18:18] <ottomata>	 i would really like to push more through eventlogging soon, so we can make use of the work we did
[16:18:26] <ottomata>	 i don't want to push a ton more all at once
[16:18:37] <ottomata>	 but i'd be fine with doubling or tripling our current throughput for your use case
[16:18:40] <ottomata>	 maybe a little mroe
[16:19:22] <ebernhardson>	 sounds reasonable, i don't actually know how many clicks we get in special:search...should probably find out
[16:19:51] <ottomata>	 ebernhardson:  we could probably ramp it up, right?
[16:19:58] <ottomata>	 start doing it sampled, and then slowly ramp it up?
[16:20:00] <ebernhardson>	 ottomata: certainly, start with a sampling and then go up
[16:20:33] <bd808>	 mforns: The logstash process is showing me that there are 11 threads in the JVM labeled "<kafka" and there is no mention of kafka in the error logs but I don't know how to dig into it deeper than that. Maybe there are ferm firewalls in prod that need to be adjusted?
[16:20:37] <ebernhardson>	 i'm also not sure what % of events we need to make a useful feedback loop, i'm mostly guessing many because our most common queries are run a couple hundred times a day, and we serve 100M requests/day
[16:20:57] <ebernhardson>	 so there is a wide variety to capture
[16:21:05] <ottomata>	 hm, mforns, bd808, there are very few EventErrors
[16:21:05] <mforns>	 bd808, 11 threads?
[16:21:26] <bd808>	 it's java. gawd knows why it spawns threads
[16:21:32] <krrrit-wm>	 (PS5) Joal: Add cassandra load job for pageview API [analytics/refinery] - https://gerrit.wikimedia.org/r/236224 (https://phabricator.wikimedia.org/T108174)
[16:21:37] <bd808>	 actually even more scary it's jruby
[16:21:40] <ottomata>	 bd808:  if you check here, can you see just EventError?
[16:21:40] <ottomata>	 http://grafana.wikimedia.org/#/dashboard/db/eventlogging?panelId=5&fullscreen
[16:22:01] <ottomata>	 it averages over a minute, so it looks like < 1 per second
[16:22:05] <joal>	 bd808: Java spawn threads to ensure its threads are running
[16:22:15] <ottomata>	 i guess that should be plenty though, about 0.5 per second is 30 per minute
[16:22:44] <mforns>	 ottomata, yes, that's the usual
[16:22:57] <bd808>	 ottomata: yeah and we haven't seen any at all into the logstash backend yet
[16:23:02] * bd808 looks at the filter again
[16:24:01] <bd808>	 hmmm.. puppet did something different in generating the config file between beta and prod
[16:24:08] <bd808>	 and it would cause it all to fail
[16:24:21] <bd808>	 "tags       => eventlogging_EventErrorkafka"
[16:24:27] <mforns>	 wow
[16:24:36] <bd808>	 instead of "tags       => ["eventlogging_EventError", "kafka"]"
[16:24:47] <mforns>	 curious
[16:25:02] <bd808>	 so there is the problem. Now I guess we need to figure out how to fix it
[16:25:11] * bd808 checks verisons
[16:25:13] <mforns>	 ok
[16:25:29] <mforns>	 brb
[16:26:13] <bd808>	 even more strangely, the beta and prod hosts have the exact same version of puppet installed
[16:27:24] <mforns>	 mmm
[16:28:23] <ottomata>	 hm
[16:28:48] <ottomata>	 hmmm, that's weird
[16:28:51] <ottomata>	 yeah, i guess if you expect tags to be an array
[16:29:05] <ottomata>	 to render it properly in the template you should stringify it
[16:29:06] <ottomata>	 mabye
[16:29:37] <ottomata>	 ["<%= @tags.join(",") %>"]
[16:29:38] <ottomata>	 ?
[16:29:42] <mforns>	 ottomata, makes sense, but the same file worked in logstash-beta, no? crazy
[16:29:53] <ottomata>	 ["<%= @tags.join('","') %>"]
[16:29:57] <ottomata>	 yeah, that is very strange
[16:30:05] <bd808>	 yeah, that would be safer I guess
[16:30:11] <ottomata>	 different versions of ruby?
[16:30:22] <bd808>	 this is all on the puppet side
[16:30:30] <ottomata>	 ja but puppet is ruby
[16:30:41] <bd808>	 oh right.
[16:30:46] <ottomata>	 .erb == embedded ruby
[16:30:53] <ottomata>	 so <%= %> blocks are ruby code
[16:31:10] <bd808>	 they should basically be the same. both are jessie and have puppet 3.7.2-4 installed
[16:31:20] <ottomata>	 hm, yeah, dunno
[16:31:26] <ottomata>	 bd808: hm
[16:31:30] <ottomata>	 what if instead of the .join
[16:31:32] <ottomata>	 you do
[16:31:36] <ottomata>	 @tags.to_s
[16:31:37] <ottomata>	 ?
[16:31:38] <ottomata>	 huh
[16:31:40] <ottomata>	 that is what <%= shoudl do.
[16:31:42] <ottomata>	 hm
[16:31:44] <ottomata>	 strange, dunno
[16:32:08] <joal>	 !log Started cassandra load jobs from 2015-10-01
[16:32:11] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[16:37:02] <mforns>	 bd808, ottomata, https://gerrit.wikimedia.org/r/#/c/244191/
[16:38:55] <ottomata>	 mforns: somehow I think int won't matter
[16:38:59] <ottomata>	 .to_s is what <%= should do.
[16:39:10] <ottomata>	 but, i guess we can try!
[16:39:10] <mforns>	 ottomata, yes
[16:39:18] <mforns>	 aha
[16:41:32] <ottomata>	 yeah, no chnage mforns
[16:41:37] <mforns>	 :/
[16:42:05] <ottomata>	 mforns:  build the string manually?
[16:42:24] <wikibugs>	 Analytics, Privacy, Varnish: Connect Hadoop records of the same request coming via different channels - https://phabricator.wikimedia.org/T113817#1709316 (GWicke) @nuria, I think @csteipp means the logs MediaWiki stores in its log tables, and not "operational" logs. There aren't any plans to store requ...
[16:42:34] <chasemp>	 ottomata: some analytics box crashed and daniel did a hard reboot I think he was pingin' you on -ops fyi
[16:45:26] <mforns>	 ottomata, looking at it
[16:55:11] <bd808>	 ottomata: did you apply that patch on the puppetmaster after you merged? If so it didn't change anything in the generated config.
[16:55:37] <bd808>	 I have a patch ready that uses join() instead
[17:03:14] <mforns>	 bd808, I think ot-tomata is having lunch, can I see the patch?
[17:03:38] <bd808>	 mforns: https://gerrit.wikimedia.org/r/#/c/244198/
[17:03:42] <mforns>	 thx!
[17:06:40] <mforns>	 bd808, have you tested this in logstash-beta? do you need me to send errors to kafka?
[17:10:11] <bd808>	 I tested it but just by comparing the prior config to the new one (they were identical)
[17:10:11] <mforns>	 bd808, aha, OK
[17:10:11] <mforns>	 bd808, hey thank you for working in that, appreciate it a lot :]
[17:10:12] <bd808>	 sure. logstash is kind of my baby so I feel obligated to help get more usage of it
[17:10:12] <mforns>	 aha, anyway thx
[17:14:42] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production {dove} [2 pts] - https://phabricator.wikimedia.org/T114881#1709457 (Fhocutt) Oh, no. Thanks for tracking this down, @mforns.
[17:28:41] <milimetric>	 ottomata: gabriel makes the valid point that "analytics" may be too abstract from the consumer's point of view.  And that "metrics" is more approachable.  Do you or anyone else feel strongly about "analytics"?  I'd like to move forward soon
[17:28:42] <milimetric>	 https://phabricator.wikimedia.org/T114830#1709209
[17:29:04] <wikibugs>	 Analytics-Kanban, Analytics-Wikimetrics: Wikimetrics' cohort page is returning 500 in production {dove} [2 pts] - https://phabricator.wikimedia.org/T114881#1709498 (egalvezwmf) Thank you!
[17:31:22] <joal>	 milimetric: no issue for me
[17:31:49] <joal>	 milimetric: issue with jobs though --> problem with keyspaces having capitals (shitty cassandra code again ...)
[17:32:17] <joal>	 milimetric: I need to go for tonight, I have a friend at home
[17:32:31] <joal>	 I'll try to solve and start the loading tomorrow asap :(
[17:36:29] <milimetric>	 joal: no worries, thx
[17:36:29] <milimetric>	 lemme know if i can help
[17:36:29] <joal>	 milimetric: I'll explain you tomorrow, it's about quoted identifiers in cassandra
[17:36:30] <joal>	 Have a good end of day all ! see y'all tomorrow :)
[17:42:02] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1709578 (Ottomata) I suggested analytics not because it is connected to the 'Analytics team' or the 'Analytics cluster', but that we have named this endpoint the Analytics Query...
[17:46:04] <ottomata>	 mforns:  bd808 looks bletter
[17:46:04] <ottomata>	 +    tags       => ["eventlogging_EventError", "kafka"]
[17:46:17] <ottomata>	 cool'
[17:46:17] <ottomata>	 https://logstash.wikimedia.org/#/dashboard/elasticsearch/eventlogging-errors
[17:46:21] <ottomata>	 here they are!
[17:46:38] <bd808>	 hey, we can haz data
[17:47:32] <bd808>	 next nice step would be getting the schema name added to the event we see in kafka
[17:47:55] <bd808>	 so we can drill down on what schema is producing bad data
[17:49:11] <wikibugs>	 Analytics, Security-Reviews: Security review of Analytics Query Service - https://phabricator.wikimedia.org/T114918#1709604 (csteipp) NEW
[17:51:40] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster - https://phabricator.wikimedia.org/T114830#1709619 (Milimetric) I agree with the wiki on what "analytics" means, and I think this particular part of the path isn't too related to the analytics query service.  We may hit t...
[17:54:15] <ottomata>	 bd808:  aye, should not be hard to do
[17:55:28] <mforns>	 bd808, ottomata, woohoo! cool!!
[17:55:41] <mforns>	 +1 schema name
[17:55:57] <mforns>	 will create a task for that
[17:55:58] <ottomata>	 ok, moving to cafe, back shortly
[17:56:25] <mforns>	 hey thank you guys for helping me *a lot* in that!
[17:56:47] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster {slug} [3 pts] - https://phabricator.wikimedia.org/T114830#1709639 (Milimetric) a:Milimetric
[18:11:10] <wikibugs>	 Analytics, Security-Reviews: Security review of Analytics Query Service - https://phabricator.wikimedia.org/T114918#1709722 (mobrovac) The main component is a #RESTBase cluster on `aqs100[123]`. It uses the vanilla version, with an additional module - [`pageviews`](https://github.com/wikimedia/restbase/b...
[18:13:01] <nuria>	 madhuvishy: interview was cacelled
[18:13:05] <nuria>	 *cancelled
[18:13:09] <madhuvishy>	 nuria: aah
[18:14:39] <madhuvishy>	 let me know if you wanna continue
[18:20:33] <nuria>	 madhuvishy: yessss
[18:20:36] <nuria>	 batcave!
[18:24:56] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster {slug} [3 pts] - https://phabricator.wikimedia.org/T114830#1709771 (GWicke) @milimetric, the new x-request-handler syntax should make this relatively straightforward to accomplish. You could use a simple proxy hierarchy wi...
[19:27:02] <nuria>	 ebernhardson: yt?
[19:28:48] <ebernhardson>	 nuria: yup
[19:29:15] <nuria>	 ebernhardson: we have had success processing records from your schema but
[19:29:36] <nuria>	 ebernhardson: the schema needed some modifications for defaults
[19:29:50] <nuria>	 ebernhardson: we are commiting a patch so you can see
[19:30:03] <nuria>	 ebernhardson: how are you validating your records against teh schema?
[19:30:09] <ebernhardson>	 there's always a but :) more defaults is probably ok.  I prefer the strictness of not having defaults, but i can see how it makes the whole thing smoother
[19:30:21] <ebernhardson>	 (and the schema upgade policy requires it)
[19:30:48] <nuria>	 ebernhardson: they do not seem to be optional
[19:31:03] <nuria>	 ebernhardson: were you using any tools to validate your records against your schema?
[19:31:33] <ebernhardson>	 nuria: in php we validate the schema with the avro library: https://github.com/wikimedia/avro-php/blob/master/lib/avro/schema.php#L389
[19:31:47] <ebernhardson>	 (it says wikimedia, but thats just because apache doesn't distribute it in a way that works for our production)
[19:32:00] <ebernhardson>	 err, validate the datum against the schema
[19:32:00] <nuria>	 ebernhardson: so if your records validated there are discrepancies between that library and the avro tools one in java
[19:32:23] <ebernhardson>	 hmm, i'm pretty sure i manually checked the json i sent you validated (i wrote it by hand, not generated by the code yet)
[19:32:33] <ebernhardson>	 lemme double check i still hvae that on my disk
[19:32:36] <nuria>	 ebernhardson: can you check again?
[19:32:48] <nuria>	 ebernhardson: sounds good cc madhuvishy
[19:34:17] <krrrit-wm>	 (PS4) Madhuvishy: [WIP] Add refinery-camus module [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990
[19:35:31] <madhuvishy>	 ebernhardson: the message was valid - the schema we had to change
[19:35:43] <nuria>	 ebernhardson: see changes to schema: https://gerrit.wikimedia.org/r/#/c/243990/3..4/refinery-camus/src/main/avro/CirrusSearchRequestSet.avsc
[19:35:54] <ebernhardson>	 madhuvishy: oh, you mean it wouldn't load the schema? i thought you meant the datum didn't match the schema
[19:35:58] <ebernhardson>	 php 100% thinks thats a valid schema :S
[19:36:06] <nuria>	 ebernhardson: it is the same problem
[19:36:15] <nuria>	 ebernhardson: teh data did not validate to schema
[19:36:20] <nuria>	 ebernhardson: we change schema
[19:36:36] <nuria>	 ebernhardson: but you could change the data too to solve teh issue
[19:36:44] <madhuvishy>	 ebernhardson: hmmm, its a bit confusing. the thing that din't work was - for union types like ["null", "int"], it expects the message to explicitly state the type
[19:36:54] <madhuvishy>	 for now we removed all the union types
[19:37:07] <madhuvishy>	 also camus expects all the fields in the schema to have defaults
[19:37:17] <ebernhardson>	 interesting, yes php thinks that the schema is valid
[19:37:18] <nuria>	 ebernhardson: makes sense?
[19:37:25] <madhuvishy>	 ebernhardson: the schema compiles
[19:37:35] <ebernhardson>	 i mean that the message matches the schema
[19:37:41] <madhuvishy>	 interesting
[19:37:49] <madhuvishy>	 do you have the avro-tools jar?
[19:37:52] <ebernhardson>	 no nulls would be annoying, but i guess thats more an anoyance for oliver and mikhail :)
[19:37:58] <nuria>	 ebernhardson: right,  different bindings
[19:38:07] <ebernhardson>	 it just means they have to do funny stuff to get averages that don't include the unknown values
[19:38:22] <madhuvishy>	 ebernhardson: there might be a way to do nulls
[19:38:27] <ebernhardson>	 like avg(null, 1, 2, 3) is not the same as avg(0, 1, 2, 3) or avg(-1, 1, 2, 3) or however you signify a "non-value"
[19:38:30] <madhuvishy>	 i am not an expert at avro schemas
[19:38:37] <ebernhardson>	 neither am i, i just read the docs one day :)
[19:38:58] <madhuvishy>	 if you have the avrotools jar, you can do
[19:39:00] <madhuvishy>	  java -jar avro-tools-1.7.6.jar jsontofrag --schema-file CirrusSearchRequestSet.avsc searchmessage.json
[19:39:03] <nuria>	 ebernhardson: bottom line is that a list [null, 1, 2, 3]
[19:39:08] <nuria>	 is not legal avro
[19:39:15] <madhuvishy>	 where the last arg is the json message you gave me
[19:39:24] <madhuvishy>	 if this throws an error, we have a problem
[19:39:35] <ebernhardson>	 ok
[19:40:22] <madhuvishy>	 apart from that, all fields should have defaults, feel free to play around with the null stuff - and if your schema validates against the tools jar, we can use it
[19:40:40] <ebernhardson>	 indeed, the java one complains "Expected start-union. Got VALUE_STRING"
[19:40:43] <ebernhardson>	 interesting
[19:40:44] <madhuvishy>	 yup
[19:40:57] <madhuvishy>	 ebernhardson: this is what i found on that http://stackoverflow.com/questions/27485580/how-to-fix-expected-start-union-got-value-number-int-when-converting-json-to-av
[19:43:04] <ebernhardson>	 hmm, i guess i need to file an upstream bug with apache for that. I've been through the code and nowhere is that taken into account in the php version of avro librar
[19:43:45] <ebernhardson>	 oh, actually i guess thats only specific to json encoding (which is unsupported by the php library, it just does binary)
[19:50:33] <ebernhardson>	 overall i think that will be fine without unions, just having the data is most important
[20:01:03] <madhuvishy>	 ebernhardson: hmmm, may be it will work with binary? i'll test that in a bit
[20:01:18] <ottomata>	 "ebernhardson: hmmm, its a bit confusing. the thing that din't work was - for union types like ["null", "int"], it expects the message to explicitly state the type "
[20:01:22] <ottomata>	 i kinda remember this, and thought it was weird
[20:01:31] <madhuvishy>	 yeah
[20:01:33] <ottomata>	 in the json encoding, its like you can't just send the value of one of the types
[20:01:39] <madhuvishy>	 yeah
[20:01:47] <ottomata>	 you actually have to send an sub object with the value and a key that specifies the type, right?
[20:01:51] <madhuvishy>	 yupp
[20:01:58] <madhuvishy>	 may be it'll be fine with binary
[20:02:00] <madhuvishy>	 i'll check
[20:13:11] <wikibugs>	 Analytics-Kanban, Analytics-Wikistats: Feed Wikistats traffic reports with aggregated hive data {lama} [8 pts] - https://phabricator.wikimedia.org/T114379#1710142 (Milimetric) a:kevinator>Milimetric
[20:18:34] <harej>	 What is the best way to get page view data for over a  thousand articles all at once?
[20:27:34] <nuria>	 harej: did you try the pageview dumps?
[20:28:15] <nuria>	 harej: for example: https://dumps.wikimedia.org/other/pagecounts-raw/2014/2014-05/
[20:28:40] <harej>	 I was just going to have a Python script query grokse 1,400 times but that reminds me that I do actually have a script for parsing through the pagecount dump.
[20:32:45] <wikibugs>	 Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1710260 (GWicke)
[20:37:40] <wikibugs>	 Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1710269 (Spage) a:Ottomata
[20:39:08] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster {slug} [3 pts] - https://phabricator.wikimedia.org/T114830#1710278 (Milimetric) @GWicke, it seems like the right way would be to factor out the docs from specs/analytics/v1/pageviews.yaml and use them in both the front-end...
[20:56:14] <mforns>	 milimetric, yt?
[20:56:21] <milimetric>	 hi mforns
[20:56:24] <mforns>	 hi!
[20:56:50] <mforns>	 I forgot to say in standup that when trying to solve the wikimetrics issue, I broke staging :/
[20:57:10] <mforns>	 I thought the problem might be with the last change from ha-shar
[20:57:22] <mforns>	 wich has to do with requirements
[20:57:29] <milimetric>	 possible yea
[20:57:36] <mforns>	 so I executed the setup.py script
[20:57:48] <mforns>	 and after that I could not recover staging
[20:57:56] <mforns>	 it seems a path problem
[20:58:08] <wikibugs>	 Analytics-Backlog: Sanitize pageview_hourly - https://phabricator.wikimedia.org/T114675#1710357 (kevinator)
[20:58:16] <milimetric>	 i'll brb, gotta help carry some stuff, but mforns don't worry about it
[20:58:30] <mforns>	 milimetric, no problem
[20:58:31] <milimetric>	 setup.py though?  did pip install -e . change?
[20:59:02] <mforns>	 I didn't try that
[20:59:30] <mforns>	 don't worry, I'll look at it, and if I can not solve it, I'll ping you aftewards
[20:59:31] <krrrit-wm>	 (CR) Nuria: "Things working now." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 (owner: Madhuvishy)
[20:59:42] <wikibugs>	 Analytics-Backlog: Sanitize pageview_hourly - https://phabricator.wikimedia.org/T114675#1702699 (kevinator)
[21:04:39] <milimetric>	 mforns: back.  If it's easier, you can just uninstall completely, wipe all the egg files and dirs, etc. and re-install
[21:11:22] <mforns>	 milimetric, I dont understand the relation between /srv/wikimetrics/dist/...egg, /srv/wikimetrics/...egg-info and /usr/local/lib/python2.7/dist-packages/wikimetrics*
[21:11:42] <milimetric>	 mforns: I don't understand that crap either, I just delete without prejudice
[21:11:57] <milimetric>	 I kind of treat broken python installations like farmers treat Avian Flu
[21:12:35] <mforns>	 xD
[21:14:39] <wikibugs>	 Analytics-Kanban, Analytics-Wikistats: Feed Wikistats traffic reports with aggregated hive data {lama} [8 pts] - https://phabricator.wikimedia.org/T114379#1710421 (ezachte) I figured we can produce all breakdowns by geography (middle column of TBD diagram) with two datasets, one for views, one for edits....
[21:36:31] <mforns>	 milimetric, wikimetrics-staging is back, thx!
[21:36:41] <wikibugs>	 Analytics-Kanban, Analytics-Wikistats: Feed Wikistats traffic reports with aggregated hive data {lama} [8 pts] - https://phabricator.wikimedia.org/T114379#1710522 (Milimetric) It's up to @Tnegrin if he wants to get geographic breakdowns, @ezachte.  I'm leaning towards handling that along with the other br...
[21:36:45] <milimetric>	 cool, thx mforns
[22:04:42] <krrrit-wm>	 (PS5) Madhuvishy: [WIP] Add refinery-camus module [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 (https://phabricator.wikimedia.org/T113521)
[22:06:01] <madhuvishy>	 nuria: ^
[22:07:51] <krrrit-wm>	 (Abandoned) Madhuvishy: [WIP] Add support to camus to consume schemaID-less avro [analytics/camus] - https://gerrit.wikimedia.org/r/243845 (https://phabricator.wikimedia.org/T113521) (owner: Madhuvishy)
[22:28:02] <nuria>	 madhuvishy: helou
[22:28:08] <nuria>	 madhuvishy: batcave
[22:28:09] <nuria>	 ?
[22:28:22] <madhuvishy>	  yup
[22:28:41] <madhuvishy>	 i'm there
[22:28:45] <nuria>	 omw
[22:32:50] <mforns>	 bye a-team! see you tomorrow!
[22:33:00] <kevinator>	 ciao!
[22:33:08] <madhuvishy>	 bye mforns :)
[22:33:12] <mforns>	 :]
[22:41:08] <wikibugs>	 Analytics-Kanban, RESTBase: configure RESTBase pageview proxy to Analytics' cluster {slug} [3 pts] - https://phabricator.wikimedia.org/T114830#1710968 (GWicke) @milimetric: yes, that ought to be possible. Only complication is that we'll have to template the URL of the actual backend service, and sub-specs...
[22:48:25] <krrrit-wm>	 (CR) Nuria: "Besides comments the only things needed is to commit our properties file we have been using to refinery side (temporarily) until these can" (2 comments) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/243990 (https://phabricator.wikimedia.org/T113521) (owner: Madhuvishy)
[22:49:24] <wikibugs>	 Analytics-Kanban, Analytics-Wikistats: Feed Wikistats traffic reports with aggregated hive data {lama} [8 pts] - https://phabricator.wikimedia.org/T114379#1710977 (ezachte) The quick survey shows most support for continuation of the geographic reports (report 21-24) , more than other breakdowns https://ww...
[23:53:54] <wikibugs>	 Analytics, Security-Reviews: Security review of Analytics Query Service - https://phabricator.wikimedia.org/T114918#1711162 (csteipp) Thanks @mobrovac. Once that code forwards the request to restbase, e.g., https://github.com/wikimedia/restbase/blob/master/mods/pageviews.js#L221, what does request look?...