[00:36:11] <James_F>	 Hey, now that https://gerrit.wikimedia.org/r/#/c/273549/ is merged into analytics/limn-edit-data, how do we get it pushed into production so it runs?
[01:27:00] <Krenair>	 is this edit-analysis.wmflabs.org ?
[01:31:14] <Krenair>	 seems to be in the 'dashiki' project in labs
[01:31:48] <Krenair>	 I would ask Yuvi, madhuvishy, or milimetric
[08:15:25] <elukey>	 hello!!
[08:15:55] <elukey>	 I just tried to run "make" in my varnish-kafka dir with libvarnish 4.0
[08:16:29] <elukey>	 and it wasn't good
[08:16:35] <elukey>	 everything has changed :D
[09:05:09] <wikibugs>	 Analytics-Tech-community-metrics, DevRel-February-2016: Backlogs of open changesets by affiliation - https://phabricator.wikimedia.org/T113719#2071554 (Qgil) How can it be that " Oldest open Gerrit changesets without code review" is from 2013 but "Oldest open Gerrit changesets by organization without code...
[10:08:27] <joal>	 !log Deploying refinery to see if previous deploy was causing https://phabricator.wikimedia.org/T128295
[10:10:36] <wikibugs>	 Analytics, Analytics-Kanban, Pageviews-API: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2071617 (JAllemandou) First investigations: The problem comes from page_title computation on the cluster. It seems the cluster...
[10:23:33] <wikibugs>	 Analytics, RESTBase: Wrong analytics for ru.wikinews.org through RESTBase - https://phabricator.wikimedia.org/T128360#2071641 (Krassotkin)
[10:25:20] <wikibugs>	 Analytics, RESTBase: Wrong analytics for ru.wikinews.org through RESTBase - https://phabricator.wikimedia.org/T128360#2071655 (JAllemandou)
[10:25:22] <wikibugs>	 Analytics, Analytics-Kanban, Pageviews-API: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2071656 (JAllemandou)
[10:27:41] <wikibugs>	 Analytics, Analytics-Kanban, Pageviews-API: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2071660 (JAllemandou) Tried to deploy refinery in hdfs to see if the problem could be coming from a bad deployed version. There...
[11:10:04] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2071748 (Aklapper) [offtopic]  >>! In T103292#2069654, @Aklapper...
[11:18:05] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2071755 (Krenair) >>! In T103292#2071748, @Aklapper wrote: >List...
[11:38:04] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2071786 (Aklapper) From a quick glance, candidate repositories to...
[11:45:16] <wikibugs>	 Analytics, Analytics-Kanban: Back-fill pageviews data for dumps.wikimedia.org to May 2015 - https://phabricator.wikimedia.org/T126464#2071792 (elukey) Managed to archive one hour of May data with:  https://wikitech.wikimedia.org/wiki/User:Elukey/Analytics/PageViewDumps
[11:55:26] <elukey>	 joal: hellooooo
[11:55:56] <elukey>	 I updated https://wikitech.wikimedia.org/wiki/User:Elukey/Analytics/PageViewDumps with my latest changes.. I am running an oozie job atm and it is filling data nicely in my tmp dir
[11:56:32] <elukey>	 and I tried to -get them and inspect with zless, looks good to me
[12:00:26] <elukey>	 (if you want to double check hdfs dfs -ls /tmp/elukey/pageviewdumps/archive/pageview/legacy/hourly/2015/2015-05)
[12:02:01] * elukey commutes to the office, brb in ~15mins
[12:11:04] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2071834 (Qgil) Looks good to me and, in fact, I thought we got ri...
[12:21:29] <elukey>	 back!
[12:40:11] <wikibugs>	 Analytics, Analytics-Kanban, Pageviews-API: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2069811 (Shizhao) Since 2016-02-23 many wiki  pageviews API encoding error, see https://wikimedia.org/api/rest_v1/metrics/pagev...
[12:59:33] <wikibugs>	 Analytics, Analytics-Wikistats, DevRel-February-2016: Clean the code review queue of analytics/wikistats - https://phabricator.wikimedia.org/T113695#2071930 (Aklapper)
[13:03:59] <joal>	 Hi elukey
[13:04:31] <joal>	 We have an issue currently on the cluster, I'll double check your code/reuslts after having solved the other one :)
[13:16:03] <joal>	 elukey: Have restarted something on the cluster in the last hour ?
[13:17:54] <grrrit-wm>	 (PS1) Joal: Change Hadoop and hive dependencies to latest CDH [analytics/refinery/source] - https://gerrit.wikimedia.org/r/273889
[13:44:11] <wikibugs>	 Analytics, Operations, Traffic: varnishkafka integration with Varnish 4  for analytics - https://phabricator.wikimedia.org/T124278#2072057 (elukey) Summary of my findings so far:  Varnish logs various kind of data (statistics, requests handled, etc..) in a shared memory file rather than in a file to a...
[13:45:28] <elukey>	 joal: sorry just seen the message! nope I haven't done anything, just kicking off an oozie job
[13:47:51] <elukey>	 joal: brb in 10 minutes for a coffee, but please let me know if I can help with the cluster :)
[14:00:23] <mforns>	 hey a-team
[14:01:34] <elukey>	 mforns: o/
[14:02:01] <mforns>	 hey :]
[14:09:24] <wikibugs>	 Analytics, Operations, Traffic: Sort out analytics service dependency issues for cp* cache hosts - https://phabricator.wikimedia.org/T128374#2072088 (BBlack)
[14:10:25] <wikibugs>	 Analytics, Operations, Traffic: Sort out analytics service dependency issues for cp* cache hosts - https://phabricator.wikimedia.org/T128374#2072101 (BBlack)
[14:15:35] <elukey>	 ottomata: https://phabricator.wikimedia.org/T124278#2072057 (work in progress)
[14:27:36] <grrrit-wm>	 (PS3) Mforns: Add hive queries for the traffic breakdown reports [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/272635 (https://phabricator.wikimedia.org/T127326)
[14:31:34] <grrrit-wm>	 (CR) Mforns: "I set the start date for the reports on June 2015, that's the first date that we have data in the intermediate browser_general table since" [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/272635 (https://phabricator.wikimedia.org/T127326) (owner: Mforns)
[14:43:06] <wikibugs>	 Analytics: Migrate limn-mobile-data/reportupdater reports to use standalone reportupdater - https://phabricator.wikimedia.org/T128375#2072136 (mforns)
[14:45:55] <grrrit-wm>	 (PS2) Mforns: Add all reportupdater files to this dedicated repo [analytics/reportupdater] - https://gerrit.wikimedia.org/r/272712 (https://phabricator.wikimedia.org/T127327)
[14:49:04] <wikibugs>	 Analytics, Operations, Traffic: varnishkafka integration with Varnish 4  for analytics - https://phabricator.wikimedia.org/T124278#2072177 (Ottomata) OH awesome!  Does the Varnish Utils lib have a similar ABI enough to just work with existing varishkafka code?
[14:51:20] <joal>	 Hi ottomata
[14:52:10] <ottomata>	 HIIII
[14:52:13] <ottomata>	 i see your email!
[14:52:18] <ottomata>	 am getting through others
[14:53:07] <joal>	 np ottomata, I subscribe you to the phab task I'm using
[14:54:01] <ottomata>	 k vool
[14:54:02] <ottomata>	 cool
[15:07:35] <ottomata>	 woah!  I didn't know phab picked up ticket IDs from SAL, cool!
[15:08:25] <ottomata>	 ok joal am looking now
[15:08:40] <ottomata>	 that upgrade in progress thing is very very strange, and actually has been with us for a long while.
[15:08:56] <ottomata>	 since some previous upgrade.  i looked into it once, but everything i could find says that the upgrade was finalized
[15:09:01] <ottomata>	 its only in the gui that it says it isn't
[15:09:09] <ottomata>	 but, bad connect ack...
[15:09:09] <ottomata>	 hm
[15:12:06] <ottomata>	 joal:  do you always get that badConnectAck?
[15:12:10] <ottomata>	 is it just while deploying to hdfs?
[15:19:26] <milimetric>	 hey ottomata & joal: how can I help with this weird encoding issue?
[15:24:01] <ottomata>	 not sure yet
[15:24:12] <ottomata>	 gotta figure out where its coming from
[15:24:20] <ottomata>	 i'm running joal's query on just the raw table
[15:24:44] <ottomata>	 pre refined using just the UDFs
[15:26:58] <milimetric>	 ok, so investigative stage.  Cool, I can do that
[15:27:52] <wikibugs>	 Analytics, Operations, Traffic: varnishkafka integration with Varnish 4  for analytics - https://phabricator.wikimedia.org/T124278#2072286 (elukey) @Ottomata not sure yet, but it would be nice to have a single source code branch.  The next step is for me and @ema to figure out if and how we could use...
[15:28:34] <wikibugs>	 Analytics, Analytics-Kanban, Pageviews-API: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2072291 (Ottomata) FYI, I just ran simliar query on just wmf_raw.webrequest table.  ``` ADD JAR /usr/lib/hive-hcatalog/share/hc...
[15:29:34] <ottomata>	 milimetric: so, it seems the GetPageviewInfoUDF
[15:29:34] <ottomata>	  UDF is returning two different things for the same input
[15:29:48] <milimetric>	 ha, ok, looking at that
[15:29:56] <ottomata>	 could be because different things on different nodes
[15:29:58] <ottomata>	 not sure
[15:34:55] <milimetric>	 ottomata: can we execute `select get_pageview_info('en.wikipedia.org', '/wiki/Lasse_%C3%85berg', '')` on each of the nodes?
[15:35:08] <milimetric>	 like, how would I do that, I can do it
[15:35:31] <ottomata>	 hmmmm
[15:35:46] <ottomata>	 hmmMmmM
[15:36:13] <ottomata>	 we'd need to dl the refinery jar with that func, compile some java code that does that, and then run it with java --classpath=refinery-core.jar
[15:36:17] <ottomata>	 but that can be done!
[15:36:43] <ottomata>	 milimetric:  if you write a simple java class with amain func that does that
[15:36:45] <ottomata>	 actually
[15:36:46] <ottomata>	 have it call
[15:36:47] <ottomata>	 pageviewDefinition.getPageTitleFromUri
[15:37:00] <ottomata>	 org.wikimedia.analytics.refinery.core.PageviewDefinition.getPageTitleFromUri
[15:37:04] <ottomata>	 i can see about doing the rest
[15:37:19] <milimetric>	 k
[15:37:30] <ottomata>	 good idea
[15:38:06] <ottomata>	 i was going to see if there was a way to do that in hive and print out the nodename
[15:38:13] <ottomata>	 that the code is executing on
[15:38:16] <ottomata>	 in the results
[15:38:19] <ottomata>	 but i'm not sure if that is possible
[15:38:20] <ottomata>	 hmMmMMM
[15:38:30] <ottomata>	 we'd have to make a UDF that got the current hostname
[15:41:56] <milimetric>	 ottomata: this work?
[15:42:02] <milimetric>	 https://www.irccloud.com/pastebin/KZbrOrrd/
[15:42:26] <milimetric>	 (not sure how to package / import)
[15:42:42] <wikibugs>	 Analytics, Analytics-Kanban, Pageviews-API: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2072347 (Ottomata) I ran this a second time, and got slightly different results, indicating to me (as @JAllemandou already note...
[15:43:03] <ottomata>	 milimetric:  that'll doi t
[15:43:05] <ottomata>	 print it out
[15:43:07] <milimetric>	 oops, I guess like print
[15:43:08] <milimetric>	 right
[15:43:11] <ottomata>	 i'll take care of linking
[15:43:27] <ottomata>	 actually,
[15:43:33] <ottomata>	 do you have refinery-core.jar somewhere?
[15:43:36] <ottomata>	 don't worry if not
[15:43:42] <wikibugs>	 Analytics, Operations, Traffic: varnishkafka integration with Varnish 4  for analytics - https://phabricator.wikimedia.org/T124278#2072350 (ema) Although it would be great for libvarnishtools to be a proper shared library, that is apparently not going to happen anytime soon in varnish itself, maybe fo...
[15:43:46] <ottomata>	 actually nm, i got it
[15:43:52] <milimetric>	 https://www.irccloud.com/pastebin/xuvmQPNY/
[15:44:01] <ottomata>	 ok trying..
[15:44:27] <milimetric>	 uh... I mean I can build refinery-core but aren't we trying to see if that's different on different nodes too?
[15:44:33] <ottomata>	 i got it
[15:44:36] <ottomata>	 hm, but hm
[15:44:38] <ottomata>	 its from hdfs
[15:44:40] <ottomata>	 so it won't be
[15:44:46] <ottomata>	 i think we are trying to see if some java lib is
[15:44:52] <milimetric>	 ah, gotcha
[15:45:12] <milimetric>	 I'll keep reading the code of the UDF and PageviewDefinition, to see if it's possible something is ambiguous
[15:45:14] <ottomata>	 k
[15:45:26] <milimetric>	 I gotta say, there are a lot of moving parts for simple string processing.
[15:45:34] <joal>	 milimetric, ottomata: sorry, Need to care Lino
[15:45:35] <milimetric>	 Like, a dailywtf-worth of moving parts :)
[15:45:46] <milimetric>	 joal: psh, get outa here, we got this
[15:46:11] <nuria>	 ottomata: did that script work? seems like it wouldn't
[15:46:14] <joal>	 milimetric: the decoding happens at the end of the getTitle stuff: using our own PercentDecoder
[15:46:30] <wikibugs>	 Analytics, Operations, Traffic: varnishkafka integration with Varnish 4  for analytics - https://phabricator.wikimedia.org/T124278#2072374 (elukey) So the plan would be to create two separate branches:  1) 3.X containing the current codebase 2) 4.X containing the vut.c/.h files to leverage the tools l...
[15:46:48] <nuria>	 ottomata: cause that is not executed through hdfs java environment
[15:47:15] <joal>	 also ottomata, hive is unusually slow (taking very long time to even launch map-reduce job --> could be related to metastore config ...)
[15:47:23] <ottomata>	 nuria:  sure it totally works
[15:47:37] <ottomata>	 just like tests work outside of hadoop
[15:47:37] <nuria>	 ottomata: executing the libs you are using on hdfs?
[15:47:45] <ottomata>	 it doesn't use any hdfs libs
[15:47:45] <joal>	 ottomata: Got the hdfs error only when trying to deploy (but didn't try anything else)
[15:48:07] * joal is back to Lino's food
[15:48:07] <ottomata>	 hmm ok joal
[15:48:08] <nuria>	 ottomata: sorry, executing all java libs that hdfs has on its classpath
[15:48:15] <ottomata>	 joal:  ja i noticed that too, takes a long time to start
[15:48:21] <ottomata>	 nuria: ?
[15:48:57] <ottomata>	 i think there is an issue with analytics1031
[15:49:12] <ottomata>	 joal, did you notice if hte IPs for that error cahnged
[15:49:13] <ottomata>	 ?
[15:49:15] <ottomata>	 or was it always .131?
[15:49:20] <nuria>	 ottomata: was wondering how do we ensure that java classpath is the same than teh one you have when running inside hadoop when executing snippet
[15:49:48] <ottomata>	 nuria:  not sure why it would matter, since it doesn't use any code from hadoop jars
[15:50:07] <ottomata>	 although, milimetric i do wonder, where is PercentDecoder from?
[15:50:11] <ottomata>	 i don't see it in import
[15:50:15] <ottomata>	 is it buildin or somethign?
[15:50:24] <milimetric>	 uh
[15:51:16] <milimetric>	 I remember jo telling me about this
[15:52:22] <milimetric>	 that doesn't seem to make sense...
[15:52:25] <nuria>	 ottomata: not hadoop libs per se but any libs in the path. It is very possible to have two versions of any package deployed and teh version you use depends on the one on classpath
[15:53:35] <milimetric>	 ottomata: does it just not need to import because it's in the same package? package org.wikimedia.analytics.refinery.core;
[15:53:46] <ottomata>	 oh
[15:53:47] <ottomata>	 then yes
[15:54:01] <ottomata>	 sorry, it just sounded so generic
[15:54:05] <ottomata>	 forgot it was custom
[15:54:05] <nuria>	 ottomata: also.. (and please discard if it seems naive) did we execute `locale` in all nodes? to make sure that is set correctly
[15:54:54] <milimetric>	 right, same package, no import, verified
[15:56:22] <milimetric>	 now I'm wondering why we made our own decoder... https://github.com/wikimedia/analytics-refinery-source/blob/405488a2d3678b5d3743b4164daa2acd2d3663e4/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PercentDecoder.java
[15:56:40] <ottomata>	  * Decoder for percent encoded strings
[15:56:41] <ottomata>	  * <p/>
[15:56:41] <ottomata>	  * In contrast to URLDecoder, this decoder keeps percent signs that are not
[15:56:41] <ottomata>	  * followed by hexadecimal digits, and does not convert plus-signs to spaces.
[15:57:09] <milimetric>	 but the normal decoder can have exceptions I think
[15:57:57] <milimetric>	 ah, no that's python, I'm getting confused
[16:01:07] <ottomata>	 hmm, there is def a problem with analytics1031
[16:01:26] <milimetric>	 you mean it's producing the weird characters?
[16:01:27] <ottomata>	 not sure if it is related to this
[16:01:55] <ottomata>	 OH metting!
[16:13:29] <wikibugs>	 Analytics, ArchCom-RfC, Discovery, EventBus, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#2072426 (Ottomata) We will resolve this after T120212 is closed, and after we have the first consumer (change propagation) in production.
[16:16:21] <James_F>	 milimetric: Hey, now that https://gerrit.wikimedia.org/r/#/c/273549/ is merged into analytics/limn-edit-data, how do we get it pushed into production so it runs?
[16:17:03] <milimetric>	 James_F: those run on puppet with ensure: latest
[16:17:11] <milimetric>	 so it should be running at the next daily run
[16:17:22] <milimetric>	 I think that's 19:00 UTC or something like that... not sure where it gets those times
[16:18:01] <milimetric>	 oh! you mean it hasn't run for 2 days... !  I'll look at it
[16:20:57] <James_F>	 milimetric: Yeah. Thanks
[16:22:05] <milimetric>	 James_F: I see, the job started on Feb. 27 but since it has a million days to catch up is taking a long time
[16:22:31] <milimetric>	 James_F: while I have you, do you really need the compare dashboard to load up by default with all the data from April?
[16:22:57] <James_F>	 milimetric: Aha.
[16:23:00] <milimetric>	 It has to transfer all the text regardless since this is based on flat files, but it's much faster if it doesn't have to render all that data, especially the first graph
[16:23:14] <James_F>	 milimetric: No, defaulting to the last three months but being able to access older data would be fine.
[16:23:19] <James_F>	 And faster is always nice. :-)
[16:23:21] <milimetric>	 k, i'll do that
[16:23:29] * milimetric presses the "faster" button
[16:23:34] <James_F>	 Awesome.
[16:24:08] <James_F>	 (Is it re-calculating its data from scratch, or 'just' from 1 December when it was switched off?)
[16:28:43] <milimetric>	 James_F: december unless something's wrong
[16:28:58] <milimetric>	 it just uses the files as the source of truth - whatever's missing it runs
[16:30:09] <James_F>	 Cool.
[16:30:09] <James_F>	 So in future it won't take forever. :-)
[16:30:47] <nuria>	 a-team: standdupp
[16:33:03] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, DBA, Patch-For-Review: Add autoincrement id to EventLogging MySQL tables. {oryx} - https://phabricator.wikimedia.org/T125135#2072462 (Nuria) Ping @jcrespo , let us know if now that edit table is trimmed we can proceed with this.
[16:36:38] <wikibugs>	 Analytics, Operations, Traffic: varnishkafka integration with Varnish 4  for analytics - https://phabricator.wikimedia.org/T124278#2072492 (Ottomata) Sounds good.  Let’s make a new 3.x branch now, and make master be for 4.x support.
[16:38:35] <wikibugs>	 Analytics, Operations, Traffic: varnishkafka integration with Varnish 4  for analytics - https://phabricator.wikimedia.org/T124278#2072493 (Ottomata) Would it be helpful to build a varnish utils .deb package instead of directly adding sources?  If so, I can help with that.
[16:39:30] <ebernhardson>	 i dunno if it means anything, but i'm trying to deploy the discovery analytics repo to hdfs, and getting data timeouts: https://phabricator.wikimedia.org/P2687
[16:39:53] <ebernhardson>	 from 10.64.36.128 and 10.64.36.131
[16:41:37] <ebernhardson>	 it eventually completed though
[16:57:41] <joal>	 ebernhardson: Hey, we have experienced similar problems, ottomata is onto it :)
[16:57:48] <ottomata>	 yeah
[16:57:52] <ottomata>	 something is wrong with 131
[16:58:02] <ottomata>	 ebernhardson:  as far as I've seen, it should work, right?
[16:58:09] <ottomata>	 its just that .131 has problems
[16:58:14] <joal>	 ottomata: ebernhardson notived 128 as well
[16:58:14] <ottomata>	 and prints errors
[16:58:21] <ottomata>	 joal:  that's in the from
[16:58:24] <ottomata>	 oh FROM
[16:58:24] <ottomata>	 ?
[16:58:32] <ottomata>	 OH HM
[16:58:33] <ottomata>	 yes
[16:58:37] <ottomata>	 gr
[16:58:38] <ottomata>	 ok
[16:58:50] <ottomata>	 ebernhardson:  i have ops meeting, but will be poking at this
[16:58:51] <ottomata>	 thank you
[16:59:05] <joal>	 Also milimetric, even if utterlyWTF, page_title extraction is not speculative :)
[17:03:16] <wikibugs>	 Analytics, Analytics-Cluster: Improve Hue user management - https://phabricator.wikimedia.org/T127850#2056054 (Milimetric) p:Triage>Normal
[17:04:44] <wikibugs>	 Analytics, Analytics-Cluster: Enable purging of old jobs and coordinators in oozie - https://phabricator.wikimedia.org/T127988#2060457 (Milimetric) p:Triage>High
[17:07:39] <wikibugs>	 Analytics, Analytics-Cluster: Use MySQL as Hue data backend store - https://phabricator.wikimedia.org/T127990#2072637 (Milimetric) p:Triage>Normal
[17:08:21] <wikibugs>	 Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2072641 (Milimetric) p:Triage>Normal
[17:10:17] <wikibugs>	 Analytics-Kanban: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2072647 (Milimetric) p:High>Unbreak!
[17:13:02] <wikibugs>	 Analytics, Operations, Traffic: Sort out analytics service dependency issues for cp* cache hosts - https://phabricator.wikimedia.org/T128374#2072088 (Milimetric) p:Triage>Normal
[17:14:37] <wikibugs>	 Analytics: Migrate limn-mobile-data/reportupdater reports to use standalone reportupdater - https://phabricator.wikimedia.org/T128375#2072136 (Milimetric) p:Triage>High
[17:17:03] <wikibugs>	 Analytics, Analytics-Cluster, Patch-For-Review: Create regular backups of Analytics MySQL Meta instance - https://phabricator.wikimedia.org/T127991#2072682 (Milimetric) p:Triage>High
[17:17:46] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Create regular backups of Analytics MySQL Meta instance - https://phabricator.wikimedia.org/T127991#2060511 (Milimetric)
[17:18:35] <wikibugs>	 Analytics, Easy: Create table in hive with a continent lookup for countries - https://phabricator.wikimedia.org/T127995#2060605 (Milimetric)
[17:19:10] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Set up Webrequest -> kafka flow in beta. - https://phabricator.wikimedia.org/T127369#2072694 (Milimetric)
[17:20:22] <wikibugs>	 Analytics: Make geowiki data safely public - https://phabricator.wikimedia.org/T127409#2072697 (Milimetric) p:Triage>Normal
[17:21:41] <wikibugs>	 Analytics, Pageviews-API: Pageviews API not updated on 2/18/2016 at 8;34 utc - https://phabricator.wikimedia.org/T127414#2072704 (Milimetric) Open>declined This is expected behavior, and clients should adjust.  The pageview API can only update as fast as we get data through a pipeline.  So it will...
[17:22:37] <wikibugs>	 Analytics, Services, Patch-For-Review: wikimedia.org/api and wikimedia.org/api/rest_v1 should redirect to the docs - https://phabricator.wikimedia.org/T118519#2072706 (Milimetric) Open>Resolved
[17:24:10] <wikibugs>	 Analytics: Make Pageview API date formats more flexible {slug} - https://phabricator.wikimedia.org/T118543#2072714 (Milimetric) Open>declined Rejecting my own request - we decided the API needs to be friendly to automatic clients, so its primary purpose is to be simple.  Those clients in turn can make...
[17:25:40] <wikibugs>	 Analytics: Spike: Investigate situation of logstash and hadoop logs - https://phabricator.wikimedia.org/T121418#2072730 (Milimetric) Open>declined
[17:27:48] <wikibugs>	 Analytics: update comScore description on report card - https://phabricator.wikimedia.org/T122059#2072741 (Milimetric) Open>Resolved
[17:28:12] <wikibugs>	 Analytics: We may be missing some more spiders when tagging pageviews {slug} - https://phabricator.wikimedia.org/T121934#2072743 (Milimetric)
[17:28:14] <wikibugs>	 Analytics: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#2072744 (Milimetric)
[17:28:50] <ottomata>	 hmmm, have been able to print decoded titles and hostnames with spark
[17:28:54] <wikibugs>	 Analytics: Create a central page in wikitech to act as a central hub so users know where to go for different types of data - https://phabricator.wikimedia.org/T122970#2072748 (Milimetric) p:Triage>Normal
[17:28:54] <ottomata>	 see it encoded weird on most hosts!
[17:29:00] <ottomata>	 e..g
[17:29:01] <ottomata>	 	$Lasse_��berg:	analytics1035.eqiad.wmnet
[17:29:01] <ottomata>	 	$Lasse_Åberg:	analytics1052.eqiad.wmnet
[17:29:13] <wikibugs>	 Analytics: Move IOS team piwiki usage to production instance - https://phabricator.wikimedia.org/T123262#2072753 (Milimetric) Open>Resolved
[17:29:20] <joal>	 ottomata: :(
[17:29:32] <joal>	 ottomata: How have you managed to get the computing host info ?
[17:30:15] <wikibugs>	 Analytics: python-mwviews does not handle unicode in titles - https://phabricator.wikimedia.org/T123200#2072758 (Milimetric) Open>Resolved
[17:32:52] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, Patch-For-Review, developer-notice: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2072770 (Aklapper) >>! In T103292#2071786,...
[17:33:25] <ottomata>	 joal:  https://gist.github.com/ottomata/028531cd9b1dcf3f7bb2#file-pt-scala
[17:34:16] <ottomata>	 gonna try with just PercentDecoder
[17:34:41] <joal>	 ok ottomata
[17:34:47] <joal>	 sounds very very weird this thing !
[17:35:37] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations, DevRel-February-2016, DevRel-March-2016, and 2 others: Check whether it is true that we have lost 40% of (Git) code contributors in the past 12 months - https://phabricator.wikimedia.org/T103292#2072780 (Aklapper)
[17:38:10] <ottomata>	 ok, isolated to PercentDecoder
[17:40:00] <joal>	 Also ottomata, camus complains about 131 bad connect as well :(
[17:40:36] <ottomata>	 IIINteresting joal
[17:40:41] <ottomata>	 with URLDecoder.decode(str, "UTF-8")
[17:40:45] <ottomata>	 instead of PercentDecoder
[17:40:51] <joal>	 ottomata: But partition dumps looks good
[17:40:52] <ottomata>	 everything it 100% cool
[17:41:12] <ottomata>	 joal:  ja, i think that is just a warning, hdfs if functioning, but ja 131 (and 128?) are misbehaving
[17:41:21] <joal>	 ottomata: can you try to encode to urf-8 before using percent decoder?n
[17:41:28] <ottomata>	 k
[17:41:30] <joal>	 ok ottomata, cool (for hdfs)
[17:41:39] <ottomata>	 with URLDecoder?
[17:41:42] <ottomata>	 or just some string encoding?
[17:41:49] <joal>	 just some string encoding
[17:41:59] <wikibugs>	 Analytics, HyperSwitch, Pageviews-API, Services: Better error messages on pageview API - https://phabricator.wikimedia.org/T126929#2072812 (Pchelolo) The error messages were improved in #hyperswitch [[ https://github.com/wikimedia/hyperswitch/pull/13 | PR ]]  We will make a new release soon, after...
[17:44:20] <Ironholds>	 ellery, apropos of nothing but I forgot to mention it at the all-hands: love the hair!
[17:44:38] <Ironholds>	 (for other people we just got out of a google hangouts meeting, I'm not just wandering around saying whatever comes to mind to random people)
[17:46:59] <joal>	 ottomata: some random guess: since hdfs issues with some machines, could be related to loading the refinery jar?
[17:47:35] <ottomata>	 hmm, joal, i doubt it, since i've already loaded that jar into my spark-shell session
[17:47:40] <ottomata>	 and i'm still getting the same results
[17:48:13] <ottomata>	 pretty sure my spark jar is not using hdfs
[17:48:19] <ottomata>	 sorry
[17:48:21] <ottomata>	 my spark session
[17:48:23] <ottomata>	 spark job8
[17:48:46] <ottomata>	 joal:  we are sure this was not happening before the upgrade?
[17:48:51] <ottomata>	 or maybe it was just not noticed?
[17:49:36] <ottomata>	 hmm, seems not
[17:49:47] <ottomata>	 yea, starts on the 23
[17:49:51] <joal>	 hm ... pretty sure is was not happening: data before the upgrade doesn't show both decoded and not_decoded values (double checking again)
[17:51:14] <joal>	 ottomata: are the 'failing to decode' machines the ones that, by random chance, would have had the logstash code ?
[17:58:47] <ottomata>	 no
[17:58:49] <ottomata>	 oh
[17:58:50] <ottomata>	 logstash?
[17:58:53] <ottomata>	 uhhh
[17:59:07] <ottomata>	 i don't think so joal, there were more than those
[17:59:09] <ottomata>	 not sure though
[17:59:56] <joal>	 ottomata: worse double checking
[18:00:03] <ottomata>	 ja
[18:00:12] <ottomata>	 joal:  i'm trying to isolate the encoding probl, will do that too
[18:00:53] <joal>	 sure
[18:01:00] <joal>	 ottomata: can I help ?
[18:01:16] <joal>	 with encoding, I mean :)
[18:02:12] <ottomata>	 hm, not sure, am runing local spark shells now on two of the differeing nodes, seeing if i can reproduce, then will get into PercentDecoder code to find problem
[18:02:53] <ottomata>	 bah!
[18:02:55] <ottomata>	 nope.
[18:02:56] <ottomata>	 HMMMMMm
[18:03:14] <ebernhardson>	 perhaps silly question ... are event logging uuid's monotonically increasing?
[18:03:24] <ottomata>	 i don't think so ebernhardson
[18:03:33] <ottomata>	 are you looking in mysql?
[18:03:40] <ottomata>	 or hdfs?
[18:03:59] <ottomata>	 ok joal interesting.
[18:04:12] <ottomata>	 local spark shell on nodes that gave different result
[18:04:24] <ottomata>	 give the same (correct) result for PercentDecoder.decode
[18:04:30] <ottomata>	 so, it looks like its only in yarn context that this happens
[18:04:31] <ottomata>	 HMmMMM
[18:04:36] <joal>	 ottomata: RIIIIGHT
[18:04:40] <joal>	 Getting closer !
[18:05:16] <ottomata>	 i looked at the expanded values of the full spark dist classpath when launching spark, and it was the same on both
[18:05:20] <joal>	 ottomata: Was about to tell you that I triple-miliionuple read thre decoder code: except for java.util.Array.copy, bug, the rest is not broken
[18:05:22] <ottomata>	 hmMm
[18:05:32] <ottomata>	 aye ok
[18:05:59] <joal>	 no speculative stuff, blah
[18:06:02] <ottomata>	 this is very strange.
[18:06:49] <ottomata>	 ok hm, i need to eat lunch
[18:07:07] <joal>	 np ottomata, will be with Lino a bit, then back
[18:07:13] <ottomata>	 will think about this during, then review marcel's code, then focus on this
[18:07:18] <ottomata>	 be back in a short while
[18:07:19] <joal>	 o
[18:10:28] <mforns>	 nuria, I will skip the unique device meeting unless you think I should go, I'll concentrate on reportupdater
[18:10:49] <nuria>	 mforns: k
[18:10:54] <mforns>	 thx!
[18:14:43] <ebernhardson>	 ottomata: doh i'm slow to respond. I'm looking in mysql
[18:15:22] <ebernhardson>	 since mysql doesn't have windowing functions i can't get a row from a group based on its min timestamp, so was thinking i could self-join against a query that pulls min(uuid)
[18:17:31] <ebernhardson>	 i suppose i can think of some test queries that will give a general idea of if it's probably increasing. will look. I was thinking it probably is because mysql indexes don't like when you add random ids and has a strong performance preference for monotonically increasing primary keys
[18:19:39] <wikibugs>	 Analytics, Developer-Relations, MediaWiki-API, Reading-Admin, and 5 others: Metrics about the use of the Wikimedia web APIs - https://phabricator.wikimedia.org/T102079#2072918 (bd808)
[18:21:07] <nuria>	 ebernhardson: to have a monotonically PK i think you need an autoincrement
[18:21:46] <nuria>	 ebernhardson: as the uuid is done for uniqness not to preserve "order"
[18:26:31] <nuria>	 ottomata: again something obvious
[18:26:37] <nuria>	 ottomata: are we starting teh jvm on yarn with any -D parameter for encoding/decoding
[18:29:00] <nuria>	 ottomata: as in  -Dfile.encoding=UTF-8
[18:48:53] <joal>	 ottomata: confirmation that the thing was not happening before upgrade
[18:50:09] <leila>	 ori: WSDM's best paper went to http://www.yichang-cs.com/yahoo/wsdm16_wholepage.pdf, neat research, indeed.
[18:56:31] <ottomata>	 aye joal ok
[18:59:12] <joal>	 ottomata: I'm getting out of ideas for that encoding issue :(
[19:01:14] <ottomata>	 me too...am doing review of marcel's thing
[19:01:17] <ottomata>	 then will get back to it
[19:01:25] <ottomata>	 we need to be able to debug
[19:01:28] <mforns>	 ottomata, :]
[19:01:28] <joal>	 Sure, sorry to bother
[19:01:32] <ottomata>	 joal: , am thinking of building a new PercentDecoder
[19:01:38] <ottomata>	 that has lots of debug prints
[19:01:44] <joal>	 Could do
[19:08:07] <ottomata>	 mforns:  am looking at reportupdater::job
[19:08:15] <mforns>	 ottomata, yes
[19:08:22] <ottomata>	 is there a case where the same $repository  and different $titles will be declared and used?
[19:08:26] <ottomata>	 like
[19:08:32] <mforns>	 yes
[19:08:51] <ottomata>	 reportupdater::job { "job1": repository => A,  }
[19:08:51] <ottomata>	 reportupdater::job { "job2": repository => A,  }
[19:08:52] <ottomata>	 lok
[19:09:04] <mforns>	 like we can have reportupdater-queries, with different query folders
[19:09:32] <mforns>	 for example: have reportupdater-queries repository have a browser folder and a geo folder
[19:09:37] <mforns>	 and then have:
[19:10:11] <mforns>	 reportupdater::job { 'browser': repository => 'reportupdater-queries' }
[19:10:23] <mforns>	 reportupdater::job { 'geo': repository => 'reportupdater-queries' }
[19:10:47] <ottomata>	 as is won't that run the same cronjob command though?
[19:11:02] <joal>	 nuria: You're idea seams very promissing :)
[19:11:18] <ottomata>	 nuria:  what is file.encoding?
[19:11:44] <mforns>	 ottomata, no, because the default value for the 'query_dir' parameter is $title
[19:11:57] <joal>	 ottomata: second answer http://stackoverflow.com/questions/12831138/specify-utf-8-encoding-for-hadoop-jobs
[19:12:09] <mforns>	 ottomata, and the cron command uses $query_dir
[19:12:12] <ottomata>	 reading, (sorry, shoulda just googled)
[19:12:22] <nuria>	 ottomata: -D parameter when JVM starts
[19:12:55] <joal>	 ottomata: Probably not an error due to upgrade, maybe an error due to puppetisation
[19:13:08] <ottomata>	 hm maybe
[19:13:17] <joal>	 I mean, nothin
[19:13:20] <ottomata>	 mforns: the command knows to do a different thing if query_path is different?
[19:13:28] <joal>	 sorry, doesn't cost a lot to try I guess
[19:13:55] <mforns>	 ottomata, yes, reportupdater stored the pid file and history file in that folder, so they are different things and can be executed in parallel
[19:13:57] <ottomata>	 loooks like it will just do update_reports.py $query_path ..
[19:14:02] <mforns>	 *stores
[19:14:17] <ottomata>	 oh, because $query_path contains configs?
[19:14:52] <mforns>	 ottomata, yes, query_path contains the query files (or script files) plus the config.yaml file that has metadata on the reports
[19:15:13] <mforns>	 and also is used by reportupdater to store temporary files (pidfile and history)
[19:15:13] <ottomata>	 mforns:  will be there an effort to convert the limng generate.py stuff to update_reports.py?
[19:15:33] <mforns>	 ottomata, https://phabricator.wikimedia.org/T128375
[19:15:50] <mforns>	 it's in prioritized
[19:15:55] <ottomata>	 great cool, so the case in init.pp is just temporary
[19:16:02] <mforns>	 ottomata, exactly
[19:16:04] <ottomata>	 l
[19:16:05] <ottomata>	 k
[19:16:46] <mforns>	 I hope we can migrate that soon
[19:17:00] <mforns>	 because we have 2 repos with the same code now :[
[19:17:14] <ottomata>	 mforns:  i think you should decouple the reportupdater clone paths from the data dir clone paths
[19:17:31] <mforns>	 ottomata, what do you mean?
[19:17:38] <ottomata>	 reportupdater { path => } can just say where to clone reportupdate code
[19:17:49] <ottomata>	 and reportupdater::job can have a path that says where to clone the data/config repo
[19:17:56] <ottomata>	 it doesn't have to be inside of $working_path
[19:18:05] <mforns>	 ottomata, makes sense
[19:18:10] <ottomata>	 'working_path' was just a concept from the statistics module to abstract /a from /srv
[19:18:22] <mforns>	 aha
[19:18:32] <joal>	 nuria: YOU WIN :)
[19:19:27] <ottomata>	 mforns:  would it make sense to keep the logs inside of the reportupdate::job path?
[19:19:31] <ottomata>	 instead of in /var/log/reportupdater?
[19:19:40] <ottomata>	 then each dir would ahve its own log file local to itself?
[19:19:42] <ottomata>	 not sure
[19:19:48] <mforns>	 fine by me
[19:19:55] <mforns>	 the output could be inside also...
[19:20:09] <ottomata>	 yeah
[19:20:16] <mforns>	 ottomata, mmmm, the output maybe not
[19:20:24] <joal>	 nuria, ottomata : https://gist.github.com/jobar/5d09f6de2ad6b40fa782
[19:20:25] <ottomata>	 how is output_path used?
[19:20:27] <nuria>	 joal: does the -Dfile fix teh encoding issue?
[19:20:40] <mforns>	 ottomata, it will help for all the jobs output to the same dir, so that it can be rsync'd
[19:20:40] <ottomata>	 i don't see anything writing to it
[19:20:46] * joal bows to nuria superpowa
[19:21:07] <mforns>	 ottomata, it just ensures that it can be written by the user
[19:21:11] * milimetric shakes his fist at the encoding gods
[19:21:17] <ottomata>	 nice!
[19:21:23] <mforns>	 ottomata, and also, if rsync is specified, it's the path that will be rsync'd
[19:21:27] <ottomata>	 but how does reporupdater know to write data there?
[19:21:39] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2073192 (madhuvishy) After some discussion, we've decided to drop ClientIPs from the even...
[19:22:10] <mforns>	 the repos holding the queries and the config.yaml <- it should be in the yaml file
[19:22:17] <ottomata>	 hm
[19:22:31] <ottomata>	 so it is configurable in the puppet module, but it is also configurable in the config.yaml?
[19:22:35] <ottomata>	 but neither know about the other?
[19:22:39] <mforns>	 ottomata, yes :/
[19:22:43] <nuria>	 joal: what i do not understand...
[19:22:55] <nuria>	 joal: is how it worked in some nodes, that has to be OS issue
[19:23:08] <nuria>	 joal: it would make sense if "locale" was different
[19:23:20] <joal>	 nuria: I must say I have no idea !
[19:23:25] <mforns>	 ottomata, the idea in generate.py was to be able to create subdirectories inside the output_folder
[19:23:52] <ottomata>	 mforns:  maybe output_path should be a parameter to ::job instad, and that should be passed to command for cron job
[19:23:59] <mforns>	 ottomata, but of course, you can break things if you configure the yaml to output outside the output_folder
[19:24:17] <joal>	 ok a-team, I need to go soon, but I'll take on me to re-run jobs from tomorrow morning to fix the mess
[19:24:25] <ottomata>	 mforns: it seems like your config yaml should,n't really know about stat1002 directory hierarchies, no?
[19:24:26] <mforns>	 ottomata, generate.py doesn't accept that input parameter
[19:24:29] <joal>	 It'll be a duty I think I can handle having Lino at home :)
[19:24:35] <mforns>	 ottomata, sure
[19:24:44] <ottomata>	 joal:  ok, but what do we need to do?
[19:24:48] <ottomata>	 this is happening in hive and spark
[19:24:56] <ottomata>	 we should figure out why this is set differently on different nodes
[19:24:57] <ottomata>	 i'll try and do that
[19:25:08] <ottomata>	 mforns:  can you link me a config.yaml again?
[19:25:18] <milimetric>	 joal: have a nice night, I'm happy if Lino launches the jobs, he's gotta start somewhere right?
[19:25:36] <mforns>	 ottomata, https://github.com/wikimedia/analytics-limn-mobile-data/blob/master/mobile/config.yaml
[19:25:36] <joal>	 indeed milimetric, oozie at first, anything else comes easy :)
[19:25:56] * milimetric wishes he was taught oozie at birth
[19:25:57] <ottomata>	 hm
[19:26:01] <ottomata>	 i see
[19:26:11] <joal>	 ottomata: I wonder if it wouldn't be better to have an explicit java option for jobs
[19:26:48] <mforns>	 ottomata, also, it would be nice to be able to specify different outputs for different reports, that's a feature we're considering... this would be via config
[19:26:56] <ottomata>	 well, woudln't hurt joal, but i want this to be correct by default too
[19:27:25] <joal>	 ottomata: http://stackoverflow.com/questions/1006276/what-is-the-default-encoding-of-the-jvm
[19:27:37] <ottomata>	 ok mforns hm
[19:27:48] <mforns>	 ottomata, but you're totally right
[19:28:08] <ottomata>	 mforns:  ok general thoughs:
[19:28:15] <ottomata>	 i'm ok with this two $output config thing for now
[19:28:15] <ottomata>	 but
[19:28:23] <mforns>	 hehe
[19:28:55] <ottomata>	 i thikn you should make reportupdater init.p  handle cloning the repo, doing file perms like you have, and then setting any variables that are totally indepdent of jobs
[19:29:06] <ottomata>	 then, make ::job do things that are related to job
[19:29:08] <joal>	 ottomata: weird though: same locale on two machines having different outcome ...
[19:29:11] <ottomata>	 i would even move the resync job to the ::job class
[19:29:18] <mforns>	 ottomata, aha
[19:29:19] <ottomata>	 and just have many different crons
[19:29:43] <mforns>	 ottomata, this would actually solve everything
[19:29:44] <ottomata>	 all init.pp should do is make it possible and easy for the ::job define to work with more specific params
[19:30:04] <mforns>	 ottomata, except we would need to change generate.py to accept output_folder
[19:30:16] <mforns>	 aha
[19:30:45] <ottomata>	 mforns:  i'm ok (if you put a big comment) saying that all reportupdater::job config.yamls on a given node must write to $output_path set on  reportupdater class
[19:30:50] <joal>	 ottomata: even better: http://stackoverflow.com/questions/2168350/java-charset-problem-on-linux
[19:31:00] <ottomata>	 maybe you can fix the .py stuff and add a param later
[19:31:03] <ottomata>	 and then remove that
[19:31:12] <mforns>	 ottomata, aha
[19:31:25] <mforns>	 OK, will look at it
[19:32:01] <ottomata>	 joal:  .getBytes( "UTF-8" );
[19:32:02] <ottomata>	 maybe?
[19:32:06] <ottomata>	 in PercentDecoder
[19:32:23] <joal>	 ottomata: no good to have that settled in code
[19:32:34] <ottomata>	 ?
[19:32:36] <joal>	 ottomata: Should be handled correctly by java
[19:32:45] <ottomata>	 indeed, hm
[19:32:45] <nuria>	 ottomata: also i *think* it is not likely to work if teh jvm doesn't have teh charset
[19:32:50] <nuria>	 set correctly
[19:32:51] <ottomata>	 hmm
[19:32:54] <mforns>	 ottomata, maybe the output_folder that is passed by puppet as a parameter to reportupdater can be the output root, and all configuration in the yaml file can be relative to that
[19:32:54] <ottomata>	 k maybe
[19:33:02] <nuria>	 ottomata: but not sure 100%
[19:33:10] <ottomata>	 mforns:  that would be cool
[19:33:28] <ottomata>	 well, you'd then have to infer the $output_path root somehow in the code
[19:33:30] <ottomata>	 not sure how you'd do taht
[19:33:51] <ottomata>	 but mforns maybe also change the name from $output_path to $output_basepath or seomthing like that
[19:33:51] <mforns>	 ottomata, no just pass it as a parameter
[19:33:55] <ottomata>	 oh ok
[19:33:58] <mforns>	 aha
[19:34:00] <ottomata>	 that'd be fine i suppose
[19:34:07] <ottomata>	 if you are making a param though
[19:34:12] <ottomata>	 it seems better to use absolute path
[19:34:15] <ottomata>	 to the job
[19:34:19] <ottomata>	 then you don't need to worry about it
[19:34:35] <ottomata>	 all the job specific details would be handled by the ::job define then
[19:34:42] <ottomata>	 including making sure the output_path existed, etc.
[19:35:01] <mforns>	 aha
[19:35:24] <ottomata>	 brb
[19:36:18] * joal gets to diner
[20:01:12] <ottomata>	 hmmm
[20:01:29] <ottomata>	 analytics1050.eqiad.wmnet:	UTF-8
[20:01:30] <ottomata>	 analytics1051.eqiad.wmnet:	US-ASCII
[20:01:32] <ottomata>	 totally bonkers
[20:01:33] <ottomata>	 why?
[20:01:41] <ottomata>	 that is the result of Charset.defaultCharset
[20:02:27] <ottomata>	 if i launch a local JVM on those two nodes
[20:02:30] <ottomata>	 both report UTF-8
[20:02:35] <ottomata>	 its only in yarn do we see US-ASCII
[20:02:50] <mforns>	 milimetric, I've looked at all current limn-x-data repos that run generate.py and the only one that has reports actually run by generate.py (not RU) is limn-edit-data. And those reports are running each hour but not generating any new data since 2015-04
[20:03:31] <mforns>	 milimetric, I was thinking of going ahead and completely removing generate.py, the puppet code will be a lot simpler
[20:05:40] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2073364 (Tbayer) CCing @asherman as this will affect the WikimediaBlogVisit_5308166 schem...
[20:14:38] <nuria>	 ottomata: that smells like classpath
[20:15:00] <nuria>	 ottomata: if you do ps -aux -fwww so you print the "whole" jvm
[20:15:14] <nuria>	 ottomata: start comand, does it match in both machines?
[20:15:32] <ottomata>	 checking
[20:17:03] <elukey>	 did you guys found the issue?
[20:17:24] <elukey>	 (I am still working on the lock managers, too many lines in the chat :P)
[20:18:08] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2073437 (asherman) Curious. Thanks for mentioning my use case :)!
[20:19:00] <ottomata>	 we are close i think elukey, default charset different for different nodemanager jvms
[20:19:03] <ottomata>	 not sure why though
[20:20:41] <ottomata>	 nuria:  100% identical classpaths
[20:20:50] <ottomata>	 i did this
[20:20:51] <ottomata>	 for d in $(ps -aux -fwww | grep nodemanager | tr ' ' '\n' | grep hadoop/conf | tr ':' ' '); do realpath $d; done | sort | uniq > /tmp/classpath.txt
[20:20:52] <ottomata>	 on both nodes
[20:20:55] <ottomata>	 and then diffed the files
[20:21:01] <nuria>	 ottomata: boy !!!!!
[20:21:32] <nuria>	 ottomata: and neither classpath has -Dfile. right?
[20:23:43] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2073453 (madhuvishy) Open>Resolved @asherman @Tbayer We can solve the WikimediaBlog...
[20:24:31] <ottomata>	 right
[20:24:31] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2073456 (madhuvishy) Resolved>Open Uhhh didn't mean to resolve it.
[20:25:10] <nuria>	 ottomata: arghhhhh
[20:25:25] <nuria>	 ottomata: and LC_ALL when you do >locale?
[20:25:27] <ottomata>	 nuria:  i think more liklely somehow the envs the jvms are running in are different, locale is toally different
[20:25:28] <ottomata>	 for htose envs
[20:25:29] <ottomata>	 exactly
[20:25:32] <ottomata>	 the are different
[20:25:33] <nuria>	 ahhhhh
[20:25:35] <ottomata>	 but
[20:25:38] <ottomata>	 only in the yarn jvm
[20:25:40] <nuria>	 ottomata: ok, that's it
[20:25:42] <ottomata>	 not when i just run a local jvm
[20:25:48] <ottomata>	 so i don't know how the yarn jvm is getting that setting
[20:26:01] <nuria>	 what about the linux prompt
[20:26:04] <nuria>	 ottomata: >locale
[20:26:08] <nuria>	 and LC_ALL?
[20:26:10] <madhuvishy>	 ottomata: when we look at mapred -log doesn't it spit out everything?
[20:26:47] <ottomata>	 nuria:  i'm getting that from linux prompt
[20:26:49] <ottomata>	 its always the same
[20:26:53] <ottomata>	 unless i actually launcha  yarn job
[20:26:58] <ottomata>	 and then do a shell exec to print that
[20:27:07] <nuria>	 and then it is different, i see
[20:27:08] <nuria>	 jajaj
[20:27:22] <ottomata>	 inside of yarn job, locale shell exec prints
[20:27:23] <ottomata>	 this
[20:27:23] <ottomata>	 https://gist.github.com/ottomata/442149ce0fe6b7d7a30d
[20:27:36] <nuria>	 ok, i have had this problem before but boy, this one is even more twisted
[20:27:40] <ottomata>	 US-ASCII is result of Charset.defaultCharset
[20:28:02] <ottomata>	 madhuvishy:  from the job?  with the setting?
[20:28:04] <ottomata>	 hm checking maybe
[20:28:14] <madhuvishy>	 ottomata: ya with the job id
[20:30:29] <nuria>	 ottomata: k 1 more idea
[20:31:01] <nuria>	 ottomata: the open jdk version we have in either?
[20:32:01] <ottomata>	 same on both nuria
[20:32:06] <ottomata>	 Version: 7u91-2.6.3-0ubuntu0.14.04.1
[20:32:08] <ottomata>	 java-1.7.0-openjdk-amd64 1071 /usr/lib/jvm/java-1.7.0-openjdk-amd64
[20:32:24] <nuria>	 ottomata: arghhh
[20:32:36] <nuria>	 ottomata: and of course both machines are 64bit right?
[20:33:08] <ottomata>	 ja
[20:33:26] <ottomata>	 madhuvishy:  no relevant info in either mapper log or yarn application log that i can see
[20:34:55] <madhuvishy>	 hmmm
[20:39:49] <ottomata>	 am poking around in jmx
[20:41:15] <ottomata>	 HMMMMM
[20:41:20] <ottomata>	 file.encoding
[20:41:24] <ottomata>	 ANSI_X3.4-1968
[20:43:53] <nuria>	 wait jmx?
[20:45:27] <ottomata>	 ja just poking around at running jvm settings
[20:45:28] <ottomata>	 but
[20:45:29] <ottomata>	 this works tooo
[20:45:35] <ottomata>	  sudo jinfo 119205 | grep file.encoding
[20:45:35] <ottomata>	 file.encoding = UTF-8
[20:45:37] <ottomata>	 then on 1051 (bad)
[20:45:40] <wikibugs>	 Analytics-Kanban: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2073499 (madhuvishy)
[20:45:41] <ottomata>	 file.encoding = ANSI_X3.4-1968
[20:45:46] <ottomata>	 HOWS IT GET THERE?!?!?!
[20:45:47] <yurik>	 milimetric, how far back does pageviews api show all-site data?
[20:46:28] <grrrit-wm>	 (PS7) Nuria: Fetch Pageview Data from Pageview API [analytics/dashiki] - https://gerrit.wikimedia.org/r/270867 (https://phabricator.wikimedia.org/T124063)
[20:47:03] <nuria>	 ottomata: sooo the file.encoding is on the jvm command?
[20:47:03] <wikibugs>	 Analytics-Kanban: Clean up Client IP related code on Eventlogging {oryx} {mole} - https://phabricator.wikimedia.org/T128408#2073520 (madhuvishy)
[20:47:20] <ottomata>	 no
[20:47:28] <ottomata>	 its just in the system properties of the running jvm
[20:47:31] <ottomata>	 has not been specified on cli
[20:47:45] <nuria>	 ahahaha
[20:47:47] <ottomata>	 or in the command started
[20:48:02] <ottomata>	 i'm going to restart nodemanager on an51 and see what happens
[20:48:40] <ottomata>	 !!!!
[20:48:40] <ottomata>	 file.encoding = UTF-8
[20:48:42] <ottomata>	 it changed!
[20:48:57] <ottomata>	 !!!!!!!
[20:48:58] <ottomata>	 :?
[20:49:08] <ottomata>	 maybe some strange chicken/egg problem during cluster install?
[20:49:14] <ottomata>	 i'm going to restart all nodemanagers
[20:49:17] <ottomata>	 1 by 1
[20:49:36] <madhuvishy>	 :/
[20:50:18] <grrrit-wm>	 (CR) Nuria: "Added some mote error checking plus tests on next patch." (6 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/270867 (https://phabricator.wikimedia.org/T124063) (owner: Nuria)
[20:50:37] <nuria>	 ottomata: nice
[20:54:12] <joal>	 Back from diner !
[20:54:16] <joal>	 Just backfilled
[20:54:35] <joal>	 If restarting nodemanagers solved the issue, it's a reasonnably cheap solution !
[20:55:02] <joal>	 It however completely surrealist that the locale gets changed in yarn
[20:55:20] <joal>	 ottomata: any news? Probably not yet)
[20:57:56] <ottomata>	 joal:  yes
[20:58:07] <ottomata>	 the jvm process did indeed have file.encoding set poorly
[20:58:17] <ottomata>	 i restarted nodemanager on one offending node sofar
[20:58:21] <ottomata>	 and it went away
[20:58:25] <ottomata>	 :o !
[20:58:31] <joal>	 ottomata: due to locale setting ?
[20:58:36] <ottomata>	 no
[20:58:36] <nuria>	 man...
[20:58:38] <joal>	 That's completely weird :S
[20:58:38] <ottomata>	 locale is fine
[20:58:41] <joal>	 k
[20:58:45] <ottomata>	 just the jvm process had file.encoding wrong
[20:58:48] <ottomata>	 the nodemanager process
[20:58:49] <joal>	 k
[20:58:50] <ottomata>	 don't know why
[20:58:57] <ottomata>	 maybe some chicken/egg problem when we isntalled?
[20:58:58] <nuria>	 NOBODY knows why
[20:59:00] <ottomata>	 and last restarted?
[20:59:01] <ottomata>	 dunno
[20:59:01] <nuria>	 man ...
[20:59:10] <joal>	 nuria: good catch ! We owe you a couple beer for that one :)
[20:59:20] <nuria>	 pufff.. to ottomata you mean
[20:59:44] <nuria>	 but boy, i even ask the guy in services which is  a java guru
[20:59:54] <joal>	 nuria: I think I wouldn't have found the file.encoding stuff
[21:00:00] <ottomata>	 pssh, no nuria you found the file.encoding thing,
[21:00:12] <nuria>	 to see if he could think of anything else besides that ...
[21:00:15] <ottomata>	 i would have been writing code to output individual byte values of the conversion
[21:00:23] <nuria>	 jajajaja
[21:05:14] <grrrit-wm>	 (CR) Nuria: [C: 2 V: 2] Add ApiAction avro schema [analytics/refinery/source] - https://gerrit.wikimedia.org/r/273556 (https://phabricator.wikimedia.org/T108618) (owner: BryanDavis)
[21:07:38] <dbrant>	 nuria: would you have a moment to help with a couple hive queries?
[21:08:08] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: Add IP field only to schemas that need it. Remove it from EL capsule and do not collect it by default {mole} - https://phabricator.wikimedia.org/T126366#2073624 (Nuria) @asherman, @Tbayer : IPs as an only measure  to count unique users is rea...
[21:11:57] <madhuvishy>	 dbrant: what are your questions? I'll help if I can
[21:12:02] <elukey>	 ottomata: not sure if related but one of the changes pushed to the analytics hosts recently was apache-commons
[21:12:44] <elukey>	 just mentioning but probably not remotely related
[21:12:57] <ottomata>	 elukey:  hmmmm
[21:13:06] <ottomata>	 possible, but dunno
[21:13:10] <elukey>	 I think it was a sec update
[21:13:12] <ottomata>	 ja
[21:13:28] <ottomata>	 it doesn't seem like a lib update would change the default jvm setting though
[21:16:41] <nuria>	 ottomata, elukey : right, it cannot as encoding is decided upon startup of jvm
[21:17:23] <elukey>	 nuria: yep yep but the package was installed before my rolling restart of all the JVMs
[21:17:28] <joal>	 ottomata: can you send an email to me when you're done with restarting the nodes ?
[21:17:34] <elukey>	 surely it is not related though
[21:18:10] <milimetric>	 mforns: good idea, much better to remove generate.py
[21:18:17] <mforns>	 milimetric, cool
[21:18:35] <joal>	 Also ottomata, could be a good idea to restart datanodes (maybe same issue ?
[21:18:53] <ottomata>	 ja but elukey we restarted all jvms as part of upgrade
[21:18:57] <ottomata>	 joal:  i have restarted all nodemanagers
[21:18:59] <ottomata>	 just finsihed
[21:19:01] <milimetric>	 yurik: it'll go back to May 2015 when we're done backfilling, but now I think august?
[21:19:08] <ottomata>	 and checked, they all say UTF-8 now
[21:19:09] <joal>	 great ottomata, double checking consistency ;)
[21:19:16] <ottomata>	 will check datanodes for setting
[21:19:52] <ottomata>	 ja many processes have this!
[21:19:53] <ottomata>	 joal!
[21:19:55] <ottomata>	 bad setting
[21:19:58] <ottomata>	 will restart datanodes one by one too
[21:19:59] <joal>	 Riiight
[21:20:11] <joal>	 I could feel that :)
[21:20:24] <joal>	 That hdfs error we had before might be related, no ?
[21:21:38] <joal>	 ottomata: Controlled with Spark - All good
[21:21:44] <joal>	 ottomata: Currently doing with Hive
[21:22:20] <ottomata>	 joal:  i think not
[21:22:22] <ottomata>	 its on a lot of nodes
[21:22:24] <ottomata>	 and that sounds different
[21:22:25] <ottomata>	 but still
[21:22:32] <ottomata>	 we will see i guess
[21:22:39] <joal>	 k
[21:25:38] <yurik>	 milimetric, awesome, thx.  I just updated the template a bit - now you can insert site's totals as well - https://en.wikipedia.org/wiki/Template:PageViews_graph
[21:26:19] <yurik>	 milimetric, for some reason the site's data shows monthly only for the past few months
[21:26:36] <yurik>	 but 8billion views does look impressive :)
[21:26:41] <milimetric>	 just saw that, trying to figure out why
[21:27:15] <elukey>	 a-team: finished working on mc1001 finally, going offline :)
[21:27:33] <mforns>	 elukey, cool, see you next monday!
[21:28:00] <elukey>	 ah mforns you are travelling right!
[21:28:04] <elukey>	 enjoy your trip!!
[21:28:05] <mforns>	 yep :]
[21:28:09] <mforns>	 thx!
[21:28:19] <milimetric>	 have a good trip!  My brother's in New Delhi right now
[21:28:27] <joal>	 mforns: Safe flight, and ENJOY :D
[21:28:35] <mforns>	 milimetric, cool! thanks
[21:28:45] <mforns>	 hehe but I'm not leaving yet today :]
[21:28:50] <joal>	 huhuh :)
[21:28:52] <mforns>	 thanks anyway!
[21:30:23] <ori>	 nuria: I won't make the meeting tomorrow; I have a conflict.
[21:33:28] <joal>	 ottomata: problem solved !
[21:33:33] <joal>	 ottomata: in hive as well :)
[21:34:05] <milimetric>	 yurik: the graph makes sense.  You're doing 180 days, so that's September 22nd.  There are only 4 complete months since then, Oct, Nov, Dec, Jan
[21:34:15] <joal>	 Ok a-team, I'm off for tonight, as said earlier, I'll restart jobs as needed to cope with the encoding issues.
[21:34:28] <milimetric>	 yurik: if you do like 400 days or something, you'll see August is the last month shown
[21:34:35] <Ironholds>	 ottomata, yt?
[21:34:37] <mforns>	 bye joal, have a nice week!
[21:34:40] <milimetric>	 (we're back-filling though, so everything will be ready at some point
[21:34:42] <joal>	 I'll send an email letting people know data is incorrect and we are currently recomputing, and will ask for people not to use the cluster becaude of backfilling
[21:34:43] <milimetric>	 )
[21:34:55] * milimetric caught jo in his parens - 1 point
[21:34:56] <joal>	 Thanks mforns :)
[21:35:13] <joal>	 :)
[21:35:36] * joal likes to be put between parenthesis :)
[21:37:09] <joal>	 Bye a-team ! See you tomorrow
[21:37:40] <madhuvishy>	 good night joal :)
[21:40:34] <ottomata>	 Ironholds:  hiyaa
[21:40:34] <ottomata>	 ja
[21:40:37] <ottomata>	 joal:  laters!!!
[21:40:40] <ottomata>	 thanks for your help
[21:43:27] <Ironholds>	 ottomata, so we have https://phabricator.wikimedia.org/T128117 - basically we're looking to transfer a cluster of scripts and a cron job that runs it to a non-person-tied account
[21:43:29] <nuria>	 ori:k
[21:43:35] <Ironholds>	 would the stats role account work well for this? how would we go about it?
[21:43:49] <ori>	 nuria: would you like my input on anything ahead of that meeting?
[21:44:25] <nuria>	 ori: nah, we can talk as things come up, if any
[21:44:38] <ottomata>	 Ironholds:  we recently created a 'analytics-search' user for other discovery related jobs
[21:44:42] <ottomata>	 ebernhardson: ^
[21:44:54] <Ironholds>	 ottomata, cool!
[21:45:09] <Ironholds>	 so does that mean I should bug erik instead of you? :D
[21:45:09] <ottomata>	 it was mainly for hadoop related stuff
[21:45:23] <ottomata>	 yes probably!  i'm not sure if you are asking to have it puppetized...or?
[21:51:14] <wikibugs>	 Analytics, Pageviews-API: Pageviews API not updated on 2/18/2016 at 8;34 utc - https://phabricator.wikimedia.org/T127414#2073823 (Alexdruk) As on https://wikitech.wikimedia.org/wiki/Analytics/PageviewAPI#Updates_and_backfilling:  "Updates and backfilling The data is loaded at the end of the timespan in q...
[21:51:42] <ottomata>	 elukey: , hmmMMM yeah actually, i think it can't be the commons upgrade
[21:51:51] <ottomata>	 a 'locale' shell exec from that process
[21:51:56] <ottomata>	 also printed out a bad locale
[21:55:03] <wikibugs>	 Analytics-Kanban: Pageviews API reporting inaccurate data for pages titles containing special characters - https://phabricator.wikimedia.org/T128295#2073845 (Ottomata) Phew, somehow many NodeManager (and, unrelatedly Datanode) JVM processes had got stuck with `file.encoding = ANSI_X3.4-1968` the last time the...
[21:56:13] <ebernhardson>	 Ironholds: hmm, maybe :)
[21:56:16] * ebernhardson scrolls p
[21:56:47] <ebernhardson>	 Ironholds: looks like the right group, file a task in ops-access-requests to have you and bearloga added. or maybe just bearloga because :(
[21:57:04] <Ironholds>	 ebernhardson, cool!
[21:57:16] <Ironholds>	 ottomata, mostly "an account to own the cron job and code that isn't mine"
[21:57:53] <ottomata>	 Ironholds:  how is the code going to get there?
[21:58:01] <ottomata>	 and how are the crons going to be set up?
[21:58:09] <ottomata>	 you going to just put them there manually?
[21:58:12] <ebernhardson>	 cron's should be done from puppet
[21:58:14] <ottomata>	 and have lots of documentation for others to find them?
[21:58:15] <Ironholds>	 ottomata, it's a git repo and crontab -e ;p
[21:58:30] <ottomata>	 ebernhardson:  yes and no, it depends on what the crons are
[21:58:39] <ottomata>	 generally we don't make researcher type folks puppetize everything they do
[21:58:44] <ottomata>	 it would be a lot of overhead
[21:58:45] <ebernhardson>	 oh, i suppose :)
[21:58:57] <bearloga>	 :P
[21:58:59] <ottomata>	 it also depends on how much do and forget you want
[21:59:22] <ottomata>	 if you want to make a 'prod' cron job that is maintained by the discover+analytics+ops, then yea, puppetized
[21:59:41] <ottomata>	 if you want to be fast and just run things yourself, then its ok to just maintain your own crons
[22:00:36] <yurik>	 milimetric, any plans to add "weekly" granularity/
[22:01:08] <yurik>	 milimetric, are you saying that months always show 1st-31st ?
[22:02:00] <ottomata>	 ebernhardson: were you able to deploy your hdfs stuff?
[22:02:08] <ottomata>	 i can't reproduce the error anymore since restarting some datanodes
[22:02:14] <milimetric>	 yurik: we have to balance usefulness with storage.  Storing weekly granularity per article would probably not be worth it.  For aggregate, sure
[22:02:16] <bearloga>	 ottomata: so will Ironholds and I be able to just `su analytics-search`?
[22:02:18] <ottomata>	 this one: https://phabricator.wikimedia.org/P2687
[22:02:26] <ottomata>	 bearloga:  probably not at the moment, but we could make that possible
[22:02:42] <yurik>	 milimetric, you don't need to store it - you can calculate it on the fly :)
[22:02:59] <milimetric>	 yurik: as for 1st - 31st, not all months have all those days, but yes the month has to be complete for its stats to be added
[22:03:11] <yurik>	 i realize that :D
[22:03:14] <ottomata>	 would need to be added to the analytics-search-users group
[22:03:22] <bearloga>	 ottomata: and then eventually whoever is Ollie's replacement
[22:03:40] <ottomata>	 welp, the description of that group is apt
[22:03:42] <ottomata>	 '    description: Group of users for managing search related analytics jobs'
[22:03:50] <milimetric>	 yurik: right, so calculating on the fly is the responsibility of the clients as we see it.  There are many in js, oython, etc, and things like weekly resolution should be added there
[22:03:59] <ottomata>	 it would also give rights to deploy hadoop related analytics discovery stuff
[22:04:12] <ottomata>	 i guess that's up to you if you guys want to grant that, but i think it would be fine
[22:04:15] <milimetric>	 that keeps the api simple and meeting the general need and allows clients to specialize however they wish
[22:04:29] <yurik>	 milimetric, wouldn't it be better on the server?  this way you can reduce the payload and it gets cached by varnish
[22:05:48] <milimetric>	 requests for weeks of data would be cashed whether there's an abstraction layer in front of it or the client provides the abstraction, either way
[22:06:48] <milimetric>	 the main point is that this api is trying to be friendly to machines and not people so much
[22:07:32] <yurik>	 milimetric, of course, just trying to make payload smaller
[22:08:26] <yurik>	 ok, prochtu
[22:08:36] <yurik>	 nvm
[22:09:25] <milimetric>	 that we' ve talked about, making the payload smaller.  We might do a v2 with a different output format.  But gz transmission shouldn't care too much about the repetition
[22:09:46] <milimetric>	 gtg, meeting soon
[22:25:40] <grrrit-wm>	 (PS3) Mforns: Add all reportupdater files to this dedicated repo [analytics/reportupdater] - https://gerrit.wikimedia.org/r/272712 (https://phabricator.wikimedia.org/T127327)
[22:35:09] <grrrit-wm>	 (CR) Mforns: "Sorry that I put the changes in the same patch as creating the new files. If this patch is OK, it can be merged directly, it has no deploy" [analytics/reportupdater] - https://gerrit.wikimedia.org/r/272712 (https://phabricator.wikimedia.org/T127327) (owner: Mforns)
[22:35:53] <wikibugs>	 Analytics-Kanban, Research-and-Data: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2073499 (leila) @madhuvishy I added the blocking task T125946. It's marked as resolved but it's not. It will be on March 8 or at the latest on March 14. I'll update this thread.
[22:35:59] <wikibugs>	 Analytics-Kanban, Research-and-Data: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2073970 (leila)
[22:36:16] <wikibugs>	 Analytics-Kanban, Research-and-Data: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2073959 (leila)
[22:36:34] <wikibugs>	 Analytics-Kanban, Research-and-Data: Remove Client IP from Eventlogging capsule {mole} - https://phabricator.wikimedia.org/T128407#2073994 (madhuvishy) @leila Thanks!
[22:38:21] <grrrit-wm>	 (PS4) Mforns: Add hive queries for the traffic breakdown reports [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/272635 (https://phabricator.wikimedia.org/T127326)
[22:43:25] <grrrit-wm>	 (CR) Mforns: "In the end, I removed the output_folder config." [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/272635 (https://phabricator.wikimedia.org/T127326) (owner: Mforns)
[22:44:41] <grrrit-wm>	 (CR) Mforns: "BTW, if this patch is OK, it can also be merged. It has no deployment dependencies." [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/272635 (https://phabricator.wikimedia.org/T127326) (owner: Mforns)
[22:53:21] <wikibugs>	 Analytics, ArchCom-RfC, Discovery, EventBus, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#2074035 (RobLa-WMF)
[23:01:56] <ori>	 is there a way to get the top most frequently requested images?
[23:01:59] <madhuvishy>	 nuria: do you know what the purpose of this assertion is? https://github.com/wikimedia/eventlogging/blob/master/tests/test_jrm.py#L42
[23:02:10] <madhuvishy>	 ori: of all time?
[23:02:26] <ori>	 madhuvishy: or the last day / week
[23:02:34] <madhuvishy>	 we don't publish mediacounts through the api yet, but we can get it through hive
[23:02:35] <ori>	 sorry, I should have clarified: a _public_ way
[23:02:40] <ori>	 right, that I know
[23:02:41] <madhuvishy>	 or dumps
[23:02:51] <madhuvishy>	 but that would mean reading the dumps
[23:02:56] <ori>	 right
[23:03:07] <madhuvishy>	 there is a task to publish media counts with api
[23:03:10] <madhuvishy>	 its in backlog
[23:03:16] <madhuvishy>	 no reason not to
[23:03:23] <madhuvishy>	 cc: milimetric
[23:03:48] <milimetric>	 fully support publishing media counts via AQS (with a similar module to the API)
[23:03:55] <milimetric>	 (is that what we're talking about?) :)
[23:04:45] <ottomata>	 ori http://dumps.wikimedia.org/other/mediacounts/
[23:04:49] <milimetric>	 right, it is, yes, let's do it.  It would be probably a couple weeks of work for a mediacounts/top type of endpoint
[23:05:39] <ori>	 I didn't know that there were media-specific dumps
[23:05:44] <ori>	 that is probably good enough
[23:13:20] <madhuvishy>	 ottomata: when we remove ips - we will still get them in the raw event correct? we just won't parse them
[23:14:39] <ottomata>	 yes, until we change varnishkafka settings
[23:15:03] <madhuvishy>	 ottomata: okay cool
[23:23:20] <ottomata>	 laters!
[23:45:47] <mforns>	 nuria, I will show up at standup tomorrow, just to give an update on today's work and sync with you on next steps on browser reports