[00:01:24] <grrrit-wm>	 (PS1) BryanDavis: Fix NetworkOriginUDF tests [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253790
[00:14:21] <grrrit-wm>	 (PS1) BryanDavis: Clean up IpUtil trusted proxy initialization [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253793
[00:30:07] <wikibugs>	 Analytics-Backlog: Pageview API: Escape double quotes in article titles - https://phabricator.wikimedia.org/T118913#1812574 (ellery) NEW
[00:33:15] <nuria>	 bd808: ok, i should have run tests, which i didn't . will make sure to that next time, i normally always do it and somehow forgot this time.
[00:33:22] <nuria>	 madhuvishy: yt?
[00:33:28] <madhuvishy>	 nuria: yeah
[00:47:38] <nuria>	 bd808: sorry, gerrit is slowwwww
[00:55:34] <nuria>	 bd808: mmm.. tests failing due to some camus test, is that happening for you?
[00:56:23] <wikibugs>	 Analytics-Backlog, Fundraising research, Research-and-Data: FR tech hadoop onboarding - https://phabricator.wikimedia.org/T118613#1812664 (Tbayer) > [ ] @Tbayer would you be able to give them a walkthrough of webrequest / pv data with example queries?  @DarTar : @JKatzWMF and I  think that's in the Ana...
[00:57:54] <bd808>	 nuria: yeah that one test never passed for me even before I started making patches
[00:58:16] <bd808>	 It looked like it was maybe a config problem of some sort (missing test table or something)
[00:58:52] <nuria>	 cc joal. ok. let me see if i can fix it
[01:07:00] <nuria>	 bd808: i see, test donot run if run from source dir
[01:07:10] <nuria>	 bd808: resources are there paths are wrong
[01:11:44] <wikibugs>	 Analytics-Backlog, Fundraising research, Research-and-Data: FR tech hadoop onboarding - https://phabricator.wikimedia.org/T118613#1812733 (madhuvishy) I'm around to help too - with access to stat1002/anything with regard to querying for the data. @DarTar as far as permits go, I see that andyrussg and a...
[01:31:28] <wikibugs>	 Analytics-Backlog: Pageview API: Escape double quotes in article titles - https://phabricator.wikimedia.org/T118913#1812780 (ellery)
[02:07:02] <wikibugs>	 Analytics, Services, operations: Wikimedia pageview API intermittently throwing HTTP 503s - https://phabricator.wikimedia.org/T118817#1812875 (MZMcBride) Very nice. Thank you all for the quick investigation and resolution!
[03:10:44] <Krinkle>	 Is it intentional that the /top/ action in the new page view API has a double-encoded 'articles' array?
[03:11:06] <Krinkle>	 It seems it is output as a object of JSON inside a string of json. Thus requiring explicit re-parsing
[03:11:14] <Krinkle>	 the other actions don't do that and just work directly.
[04:48:53] <Krinkle>	 Also, I'm not sure what to think of there being over 1M page views for enwiki Special:BlankPage
[04:49:02] <Krinkle>	 #3 of all page views
[05:06:35] <wikibugs>	 Analytics-Backlog: Blogpost  Pageview API - https://phabricator.wikimedia.org/T118866#1812993 (MZMcBride) This task feels like a possible duplicate of {T118471}.
[05:24:53] <wikibugs>	 Analytics, Services: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1813007 (MZMcBride) NEW
[05:41:01] <wikibugs>	 Analytics, Services: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813033 (MZMcBride) NEW
[06:57:19] <grrrit-wm>	 (CR) Gergő Tisza: Add UDF for network origin (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253046 (https://phabricator.wikimedia.org/T118592) (owner: BryanDavis)
[07:48:26] <wikibugs>	 Analytics, Services: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813150 (Tbayer) To mention other recent pageview weirdness tickets (may or may not be related, in any case they illustrate it might not be the API's fault): T117945 (https...
[09:10:15] <grrrit-wm>	 (PS1) Addshore: Retry identica metric on empty page [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253854
[09:10:28] <grrrit-wm>	 (CR) Addshore: [C: 2 V: 2] Retry identica metric on empty page [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253854 (owner: Addshore)
[09:19:46] <grrrit-wm>	 (PS1) Addshore: Add single php methdo for curl wrapper [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253855
[09:20:07] <grrrit-wm>	 (PS2) Addshore: Add single php methdod for curl wrapper [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253855
[09:20:37] <grrrit-wm>	 (CR) Addshore: [C: 2 V: 2] Add single php methdod for curl wrapper [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253855 (owner: Addshore)
[09:26:00] <grrrit-wm>	 (PS1) Addshore: Unify useragents [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253856
[09:26:13] <grrrit-wm>	 (CR) Addshore: [C: 2 V: 2] Unify useragents [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253856 (owner: Addshore)
[09:30:07] <grrrit-wm>	 (Abandoned) DCausse: Test do not merge [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253393 (owner: DCausse)
[09:31:20] <wikibugs>	 Analytics-Backlog: Create a dedicated hive table with pageview API only requests for reporting - https://phabricator.wikimedia.org/T118938#1813225 (JAllemandou) NEW
[09:48:07] <wikibugs>	 Analytics, Analytics-Kanban, Services: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813246 (JAllemandou) a:JAllemandou
[09:50:20] <wikibugs>	 Analytics, Analytics-Kanban, Services: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813252 (JAllemandou) Quick data check:   ---- Elio Motors page seems Ok ``` https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikiped...
[10:04:30] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Thqanks for the quick fix :)" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253790 (owner: BryanDavis)
[10:50:18] <wikibugs>	 Analytics-Cluster, Analytics-Kanban: {slug} Pageview API - https://phabricator.wikimedia.org/T101792#1813341 (Tgr) Duplicate of T44259?
[11:03:17] <wikibugs>	 Analytics, Analytics-Kanban, Services: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813385 (JAllemandou) Seems that this page have been widely crawled on the given days: ``` https://wikimedia.org/api/rest_v1/metrics/pageviews/per-arti...
[11:30:40] <wikibugs>	 Analytics, Analytics-Kanban, Services: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813409 (JAllemandou) And from hive, all those calls where made from a single Ruby program.  I have sent an email to the analytics team to discuss weit...
[11:49:57] <wikibugs>	 Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session:  Pageview API  overview - https://phabricator.wikimedia.org/T112956#1813435 (HYanWong) Monthly or yearly good for me too. But what might also be good (and require thought) is some measure that can be used to normalise the result...
[12:38:46] <wikibugs>	 Analytics-General-or-Unknown, WMDE-Analytics-Engineering, Wikidata, Story: [Story] Statistics for Special:EntityData usage - https://phabricator.wikimedia.org/T64874#1813479 (Addshore)
[13:50:11] <wikibugs>	 Analytics, Analytics-Kanban, Services: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813562 (Milimetric) I'm leaning towards removing spiders from top data.  It doesn't seem of import to people looking for the "popular" pages.
[14:05:31] <halfak>	 o/ joal
[14:05:37] <halfak>	 Bummer about the job failing again.
[14:06:25] <wikibugs>	 Analytics-Backlog, Services: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1813571 (Milimetric)
[14:09:14] <wikibugs>	 Analytics-Backlog, Services: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1813576 (Milimetric) Two question for y'all.  Assume we changed this:  1. Does the breaking format change bother you as consumers?  It's kind of early and if the answer...
[14:31:38] <wikibugs>	 Analytics-Backlog: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1813612 (Milimetric)
[14:31:49] <wikibugs>	 Analytics-Kanban: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813613 (Milimetric)
[14:34:30] <halfak>	 joal, what do you think about trying my crazy streaming idea?
[14:52:57] <milimetric>	 joal: so it looks like we have overwhelming +1s for monthly per-article data
[14:57:32] <wikibugs>	 Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session:  Pageview API  overview - https://phabricator.wikimedia.org/T112956#1813657 (Bluerasberry) I use this tool continuously.  http://wikipediaviews.org/  It has some problems for me and the way that I use it. I would love to see the...
[15:00:03] <wikibugs>	 Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session:  Pageview API  overview - https://phabricator.wikimedia.org/T112956#1813665 (Milimetric) @HYanWong:  > (a) the hits for the previous month, perhaps  When we add monthly granularity to the per-article endpoint this would be easil...
[15:05:30] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1813671 (Milimetric) @Legoktm: there is a client-side limitation of 2000 bytes because events come as GETs t...
[15:13:33] <wikibugs>	 Analytics-General-or-Unknown, WMDE-Analytics-Engineering, Wikidata, Story: [Story] Statistics for Special:EntityData usage - https://phabricator.wikimedia.org/T64874#1813683 (Addshore) Daily would be good. Grouped by format. We can probably ignore /entity/ as they should all redirect to Special:Ent...
[15:21:08] <mforns>	 hi a-team
[15:21:15] <milimetric>	 morning mforns
[15:21:22] <mforns>	 hi milimetric
[15:21:31] <milimetric>	 hey the varnish EL client side limit thing went up from 1000 to 2000 right?
[15:21:43] <mforns>	 milimetric, that's what I understoon
[15:21:45] <ottomata>	 hiya
[15:21:47] <mforns>	 *understood
[15:21:51] <ottomata>	 yeah somethign like that
[15:21:55] <mforns>	 from 1014 to 2000
[15:22:23] <milimetric>	 k, cool
[15:22:30] <milimetric>	 but server side there's no real limit
[15:22:53] <mforns>	 aha
[15:22:59] <mforns>	 well, there must be some limit :]
[15:23:11] <milimetric>	 well, the mysql varchar limit, whatever that's set to
[15:23:16] <milimetric>	 but i saw that insert with truncation just fine
[15:23:22] <milimetric>	 so it won't break, it'll just cut off
[15:23:25] <ottomata>	 fyi, eventlogging in beta labs is down atm, am bringing it back up shortly...
[15:23:29] <milimetric>	 thx
[15:23:37] <mforns>	 ok
[15:24:10] <mforns>	 milimetric, btw, madhuvishy sent me an email saying you were talking yesterday about where to place the wikimetrics roll up code
[15:24:33] <mforns>	 I put it inside MultiProjectMetricReport
[15:24:43] <milimetric>	 https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/jrm.py#L77
[15:24:46] <milimetric>	 191 bytes
[15:24:50] <milimetric>	 sorry, 191 characters
[15:25:00] <milimetric>	 mforns: yes
[15:25:00] <mforns>	 I was in doubt between this and creating another ReportNode class
[15:25:10] <milimetric>	 right, we should all talk when she's around
[15:25:17] <milimetric>	 make sure we're on the same page
[15:25:31] <mforns>	 ok
[15:25:36] <milimetric>	 the raw data for what we need comes out of MultiProjectMetricReport
[15:25:57] <milimetric>	 and there could either be another node on top of that like AggregateByUserNameReport
[15:26:16] <milimetric>	 she already made a top level custom node called RunGlobalReport
[15:26:29] <mforns>	 what I did is adding a parameter to MultiProjectMetricReport called group_by_user
[15:26:39] <mforns>	 and if that is true, the finish method applies the grouping
[15:26:41] <milimetric>	 oh I see
[15:26:55] <milimetric>	 and you can pass in arbitrary ways to aggregate there?
[15:27:21] <milimetric>	 for the two examples I saw, the results would have to be "OR"-ed within each user's group of projects
[15:27:31] <milimetric>	 but I'm not sure if that applies universally
[15:27:32] <mforns>	 that was the other question: you were talking about hardcoding the aggregation strategy depending on the metric
[15:28:03] <mforns>	 I don't think so
[15:28:18] <mforns>	 there are also averages and sums no?
[15:28:25] <milimetric>	 per user?
[15:28:49] <milimetric>	 so there's "Existing Active Editor" - that's OR, "Existing Users" and "New Users" - those are all "OR"
[15:28:56] <milimetric>	 lemme look at the other ones
[15:29:10] <mforns>	 I think so, when you run bytes edited per user for example
[15:32:22] <milimetric>	 it seems there's some misunderstanding
[15:32:31] <milimetric>	 they don't want to be able to run arbitrary metrics
[15:32:41] <milimetric>	 they have 4 specific metrics that always run
[15:32:49] <mforns>	 ok
[15:32:53] <milimetric>	 they're linked here in the parent bug: https://docs.google.com/spreadsheets/d/1Qib7Nm0eyE9oMrst4cR5lHLvEg5C2P5TnpYnZhOWt4w/edit#gid=1338730435
[15:33:15] <mforns>	 mmmmmm, I remember now
[15:33:24] <milimetric>	 so when they run a "global" report, they're just looking for 4 numbers as the output
[15:33:42] <milimetric>	 but the aggregation has to take into account users across wikis in certain cases
[15:33:48] <milimetric>	 so, here's what we have
[15:34:05] <milimetric>	 for those first 3 that I mentioned, we get results per user / per project
[15:34:19] <milimetric>	 we OR the results of each user across all their projects, so we have one row per user
[15:34:20] <mforns>	 yes
[15:34:31] <milimetric>	 then we SUM the output from that, and that's the number
[15:34:32] <mforns>	 gotcha
[15:34:41] <milimetric>	 for that third one, I don't think we need any custom code
[15:35:02] <mforns>	 you mean the fourth?
[15:35:04] <milimetric>	 but I just saw some contradiction on that page and some notes about doing something else "in the future".  So we should circle back with Amanda today
[15:35:07] <milimetric>	 sorry, fourth
[15:35:10] <mforns>	 ok
[15:35:26] <milimetric>	 and something about "manually adding the articles improved" or something which also doesn't make sense
[15:49:02] <joal>	 Hi guys
[15:49:11] <joal>	 halfak: wanna talk for a minute?
[15:49:30] <mforns>	 hi joal
[15:49:35] <halfak>	 o/ joal
[15:49:40] <halfak>	 batcave or IRC?
[15:49:56] <joal>	 batcave :)
[15:50:36] <joal>	 You'll be the first person I interact with since 6 hours, so let's talk :)
[15:50:44] <joal>	 halfak: --^
[15:50:44] <joal>	  :)
[15:52:13] <wikibugs>	 Analytics-Backlog, Discovery, Reading-Infrastructure-Team: Determine proper encoding for structured log data sent to Kafka by MediaWiki - https://phabricator.wikimedia.org/T114733#1813795 (Ottomata) Yeah, I'm unsure of what we should do at this point.  I won't have a lot of time to work on Avro support...
[15:52:19] <mforns>	 milimetric, "Qgil moved this task to On track on the Wikimedia-Developer-Summit-2016 workboard."
[15:52:27] <milimetric>	 I saw :)
[15:52:33] <mforns>	 :]
[15:52:34] <milimetric>	 I was waiting for standup to \o/
[15:58:31] <nuria>	 hola a-team
[15:58:38] <milimetric>	 morning nuria
[15:58:42] <mforns>	 hola!
[16:04:55] <halfak>	 milimetric, I don't have the wiki-research-l password since the mailman-pocalypse
[16:05:01] <halfak>	 So I'm waiting on DarTar
[16:05:15] <halfak>	 You might just send the message again and we can delete the old one from the queue.
[16:08:47] <wikibugs>	 Analytics-Backlog, Fundraising research, Research-and-Data: FR tech hadoop onboarding - https://phabricator.wikimedia.org/T118613#1813841 (Nuria) @DarTar: Regarding hive tutorials, we can help as @madhuvishy said. Now, I would ask that before anything they read slides of our prior classes: http://commo...
[16:09:48] <wikibugs>	 Analytics-Backlog: Blogpost  Pageview API - https://phabricator.wikimedia.org/T118866#1813842 (Nuria)
[16:09:50] <wikibugs>	 Analytics-Backlog: Write pageview API blogpost - https://phabricator.wikimedia.org/T118471#1813843 (Nuria)
[16:17:39] <nuria>	 joal: tests work fine, ignore my e-mail cc bd808
[16:17:51] <joal>	 np nuria :)
[16:17:52] <nuria>	 bd808: let me know if you still have problems
[16:18:51] <nuria>	 milimetric: can you give me today a 10 minute talk of  wikimetrics new feature?
[16:19:01] <nuria>	 milimetric: whenever is good for you
[16:20:49] <milimetric>	 nuria: yes, I'm talking to madhuvishy and mforns after standup about it
[16:20:59] <milimetric>	 if you want you can stick around and I'll do a quick summary before
[16:21:07] <milimetric>	 we get into it
[16:21:23] <grrrit-wm>	 (PS1) Addshore: Track propertycreators and bots [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253923
[16:21:41] <grrrit-wm>	 (CR) Addshore: [C: 2 V: 2] Track propertycreators and bots [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253923 (owner: Addshore)
[16:23:49] <grrrit-wm>	 (PS1) Addshore: Set bots and propcreator scripts +x [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253924
[16:24:01] <grrrit-wm>	 (CR) Addshore: [C: 2 V: 2] Set bots and propcreator scripts +x [analytics/limn-wikidata-data] - https://gerrit.wikimedia.org/r/253924 (owner: Addshore)
[16:25:52] <wikibugs>	 Analytics-Kanban: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813863 (Ironholds) It's of absolutely no import; in fact, it makes things less usable. I'd already filed a task on this: T117343
[16:29:12] <wikibugs>	 Analytics-Kanban: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813879 (Nuria) >I'm leaning towards removing spiders from top data. It doesn't seem of import to people looking for the "popular" pages. Known spiders yes, I agree, they are hard...
[16:30:31] <wikibugs>	 Analytics-Backlog: Remove spider traffic from "top" results - https://phabricator.wikimedia.org/T117343#1813880 (Nuria)
[16:31:53] <nuria>	 milimetric: yes, please, let me get teh context so i can look at tasks and do Crs
[16:32:43] <milimetric>	 k.  It might be useful to read the parent task: https://phabricator.wikimedia.org/T117285
[16:32:58] <wikibugs>	 Analytics-Backlog, Fundraising research, Research-and-Data: FR tech hadoop onboarding - https://phabricator.wikimedia.org/T118613#1813885 (DarTar) @atgo copying you as Katie is on vacation, see first item in the task description.
[16:36:26] <bd808>	 "<  Krinkle>	Also, I'm not sure what to think of there being over 1M page views for enwiki Special:BlankPage" -- That page is used for icinga monitoring -- https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/hieradata/common/lvs/configuration.yaml;22535beaaf457f284a43ecc947077cd6ff76c594$128
[16:38:09] <nuria>	 bd808: you can file a bug but easiest would be to submit a fix.See: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java#L77
[16:38:41] <nuria>	 bd808: our pageview numbers (since the beginning of time) have suffered from being affected by self inflicted traffic cc krinkle
[16:38:44] <ottomata>	 !log deploying and installing eventlogging master on eventlog1001 from new eventlogging python repo (this does not change code, just repo from which it is installed)
[16:38:51] <Krinkle>	 bd808: Is that local server health monitoring or does that go through the outside varnish layers?
[16:39:16] <nuria>	 Krinkle: well, if it ends up on webrequest table goes through varnish
[16:39:48] <Krinkle>	 I can imagine 1M requests in total from each apache to itself for icinga, but 1M in total  on 1 day to varnish seems odd.
[16:40:06] <Krinkle>	 Though I guess each varnish also has monitoring, hm..
[16:41:21] <wikibugs>	 Analytics-Backlog: Remove Special:BlankPage from pageviews  - https://phabricator.wikimedia.org/T118958#1813899 (Nuria) NEW
[16:41:34] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, EventBus, Patch-For-Review: Deploy eventlogging from new repository. - https://phabricator.wikimedia.org/T118863#1813906 (Ottomata) Done in prod.  Now to remove the deployment for the old eventlogging/EventLogging mw extension repo.
[16:42:17] <nuria>	 bd808: blank page is there: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/PageviewDefinition.java#L77
[16:42:25] <nuria>	 bd808: those chnages are not deployed though
[16:42:48] <wikibugs>	 Analytics-Backlog: Remove Special:BlankPage from pageviews - https://phabricator.wikimedia.org/T118958#1813913 (Nuria) Open>Resolved
[16:42:59] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1813915 (Legoktm) OK. At what length should we start truncating then?
[16:43:00] <wikibugs>	 Analytics-Backlog: Remove Special:BlankPage from pageviews - https://phabricator.wikimedia.org/T118958#1813899 (Nuria) This just needs deployment.
[16:48:18] <wikibugs>	 Analytics-Backlog: Build a public form that can hit the new API {kudu} [8 pts] - https://phabricator.wikimedia.org/T117289#1813927 (Nuria) A form hitting an API seems strange. Also if we are querying realtime result calculation is asynchronous, suited for an ajax ui but not a simple form (maybe this is what y...
[16:52:29] <wikibugs>	 Analytics-Kanban: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813942 (Ironholds) Should we read that as "we're working on this but generally we'll wait a while" or "we'll wait a while to fix this one until other bugs have the opportunity to...
[17:02:17] <wikibugs>	 Analytics-EventLogging, Analytics-Kanban, EventBus, Patch-For-Review: Puppetize eventlogging-service - https://phabricator.wikimedia.org/T118780#1813969 (Ottomata) Currently running eventlogging-service on deployment-eventlogging04 in beta labs deployed using trebuchet and configured using puppet a...
[17:04:43] <wikibugs>	 Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session:  Pageview API  overview - https://phabricator.wikimedia.org/T112956#1813985 (Nettrom) I'll add a +1 to wanting monthly data for a bunch of articles. Could enable some interesting changes to SuggestBot's algorithms.
[17:25:48] <wikibugs>	 Analytics-Backlog: Backfill data on cassandra removing spiders from top endpoint - https://phabricator.wikimedia.org/T118972#1814117 (Nuria) NEW
[17:26:15] <wikibugs>	 Analytics-Backlog: Remove spider traffic from "top" results - https://phabricator.wikimedia.org/T117343#1814125 (Nuria)
[17:31:56] <wikibugs>	 Analytics-Backlog: Remove spider traffic from "top" results - https://phabricator.wikimedia.org/T117343#1814147 (Nuria) * Special:BlankPage  will not be consider a "true" page going forward; https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/...
[17:32:09] <wikibugs>	 Analytics-Backlog: Remove spider traffic from "top" results - https://phabricator.wikimedia.org/T117343#1814148 (Nuria) p:Triage>High
[17:32:41] <wikibugs>	 Analytics-Backlog: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1814150 (Nuria) p:Triage>High
[17:33:01] <wikibugs>	 Analytics-Backlog: Remove spider traffic from "top" results - https://phabricator.wikimedia.org/T117343#1814151 (Ironholds) Actually IIRC from looking at that the mass of traffic lacked a UA, so eliminating automata and spiders should do it.
[17:33:43] <wikibugs>	 Analytics-Backlog: Pageview API: Escape double quotes in article titles - https://phabricator.wikimedia.org/T118913#1814155 (Milimetric)
[17:33:44] <wikibugs>	 Analytics-Backlog: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1814156 (Milimetric)
[17:43:52] <wikibugs>	 Analytics-Backlog: Remove spider traffic from "top" results - https://phabricator.wikimedia.org/T117343#1814188 (JAllemandou)
[17:43:54] <wikibugs>	 Analytics-Kanban: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1814190 (JAllemandou)
[17:45:12] <wikibugs>	 Analytics-Kanban: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1813033 (JAllemandou) I'll start filling new data with user-only top tonight (meaning data from 2015-11-18 onward will be user-only). We'll then backfill older data later.
[17:50:16] <wikibugs>	 Analytics-Backlog, WMDE-Analytics-Engineering, Wikidata, Graphite: Create a Graphite instance in the Analytics cluster - https://phabricator.wikimedia.org/T117732#1814217 (Addshore) p:Triage>Low
[17:50:38] <nuria>	 ebernhardson, dcausse : let's have a short meeting/hangout to talk about how to proceed with avro-camus json?
[17:51:09] <nuria>	 dcausse: yt?
[17:51:15] <dcausse>	 nuria: yes
[17:51:17] <ebernhardson>	 nuria: sure, now?
[17:51:28] <nuria>	 in an hour or so?
[17:51:40] <ebernhardson>	 i can do that, but it's getting late for dcausse. up to him
[17:51:54] <dcausse>	 let me check
[17:52:06] <nuria>	 dcausse: right, you let us know, it can also be early tomorrow (like 5pm your time))
[17:52:14] <dcausse>	 I'm alone with kids in 1 hour and it's dinner time, better in 2 hours I think
[17:52:55] <nuria>	 dcausse: let's do tomorrow then, right? so you do not feel rush
[17:53:06] <dcausse>	 nuria: no worry we can do it today
[17:53:18] <dcausse>	 let's say 8pm UTC?
[17:53:34] <nuria>	 ok, will set it up , let me look at ottomata's schedule
[17:54:45] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1814232 (Sadads) My gut instinct is the main use cases need the 300-400 character range: most tracking tools...
[17:57:15] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1814241 (Sadads) @Halfak can you think of a use case for anything more extensive than 400 characters? Invest...
[17:58:17] <wikibugs>	 Analytics-Backlog: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1814245 (MZMcBride) >>! In T118931#1813576, @Milimetric wrote: > Two question for y'all.  Assume we changed this: >  > 1. Does the breaking format change bother you as consumers?  I...
[17:59:16] <milimetric>	 mforns: batcav?
[18:07:30] <addshore>	 Hi all! Will it ever be possible to do something like this https://wikimedia.org/api/rest_v1/metrics/pageviews/top/wikidata/all-access/2015/11 ? (ie. not specify a specific day?)
[18:07:56] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1814297 (Halfak) @gpaumier did some work to extract URLs that appear in citations.  I wonder if we could pro...
[18:09:46] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1814303 (Halfak) ``` explain externallinks; +-------------------+------------------+------+-----+---------+-...
[18:13:45] <joal>	 !log restarted Cassandra Load - top articles daily ooziw job
[18:14:22] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1814310 (Halfak) I suppose we can use the `externallinks` table to answer this question.    ``` > select LEN...
[18:19:32] <Krinkle>	 milimetric: To clarify on https://phabricator.wikimedia.org/T118931, right now the example posted there results in a fatal exception from json parsing. The code for pagview in restbase uses some kind of primitive quote-escaping logic instead of json-encoding logic – which fails to account for strings containing quotes themselves.
[18:19:44] <ottomata>	 nuria:  fyi am not working on friday
[18:19:52] <milimetric>	 yeah, Krinkle I saw that afterwards
[18:19:55] <milimetric>	 we'll have to fix that
[18:19:57] <Krinkle>	 So even if we keep the json-inside-json thing, we should still fix that
[18:19:59] <milimetric>	 right
[18:20:20] <ottomata>	 oh i see meeting, for today, that is good
[18:20:23] <milimetric>	 that's def. a bug
[18:20:29] <Krinkle>	 though I would recommend breaking it and doing it properly. It's early enough. This is why RESTBase as versions and support levels.
[18:20:32] <Krinkle>	 It's experimental.
[18:20:42] <Krinkle>	 The longer we wait, the more damage.
[18:20:56] <gwicke>	 Krinkle++
[18:24:19] <nuria>	 ottomata: ok, set that avro meeting for today 3pm your time
[18:24:26] <nuria>	 ottomata: at dcausse 's request
[18:25:52] <ottomata>	 that is good
[18:29:41] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1814339 (Sadads) @Halfak: the useful bit of that url though, is the id= and pg= variables: so all of the ana...
[18:36:32] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1814343 (Halfak) I'm not sure that truncation is an effective strategy to prevent collecting PI -- especiall...
[18:38:11] <milimetric>	  Krinkle agreed, cool.  Thx
[18:49:01] <wikibugs>	 Analytics-Backlog: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1814415 (Addshore) I would prefer this be fixed asap :/
[18:50:13] <wikibugs>	 Analytics, CirrusSearch, Discovery, operations, audits-data-retention: Delete logs on stat1002 in /a/mw-log/archive that are more than 90 days old. - https://phabricator.wikimedia.org/T118527#1814417 (Ottomata) Hey wait a minute, isn't this already happening?  https://github.com/wikimedia/operat...
[19:03:46] <mforns>	 milimetric, madhuvishy: I want to create a page in wikitech to hold documentation for the Wikimetrics feature
[19:04:11] <mforns>	 just checking if you already created something, to avoid duplicates
[19:04:29] <madhuvishy>	 mforns: no I haven't created anything
[19:04:35] <mforns>	 ok
[19:05:20] <wikibugs>	 Analytics-Backlog, Analytics-Kanban, operations, Monitoring, Patch-For-Review: Migrate reqstats icinga alerts to new graphite metrics and deprecate or adapt reqstats gdash - https://phabricator.wikimedia.org/T118979#1814440 (Ottomata) NEW a:Ottomata
[19:05:46] <wikibugs>	 Analytics-Backlog, Analytics-Kanban, operations, Monitoring, Patch-For-Review: Migrate reqstats icinga alerts to new graphite metrics and deprecate or adapt reqstats gdash - https://phabricator.wikimedia.org/T118979#1814440 (Ottomata) a:Ottomata>None
[19:09:10] <mforns>	 madhuvishy, milimetric: https://wikitech.wikimedia.org/wiki/Analytics/Wikimetrics/Global_metrics (empty)
[19:09:50] <madhuvishy>	 Cool thanks
[19:17:09] * joal is gone for diner
[19:44:33] <milimetric>	 thx mforns
[19:44:47] <mforns>	 np :]
[19:58:43] <grrrit-wm>	 (PS1) BryanDavis: Fix javadoc comment for NetworkOriginUDF [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253958
[19:59:24] <grrrit-wm>	 (CR) Nuria: [C: 2 V: 2] Fix javadoc comment for NetworkOriginUDF [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253958 (owner: BryanDavis)
[20:00:59] <grrrit-wm>	 (CR) BryanDavis: Add UDF for network origin (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/253046 (https://phabricator.wikimedia.org/T118592) (owner: BryanDavis)
[20:07:02] <wikibugs>	 Analytics-Backlog: Track stats for outreach.wikimedia.org in pageview_hourly - https://phabricator.wikimedia.org/T118987#1814657 (Milimetric) NEW
[20:23:06] <wikibugs>	 Analytics, CirrusSearch, Discovery, operations, audits-data-retention: Delete logs on stat1002 in /a/mw-log/archive that are more than 90 days old. - https://phabricator.wikimedia.org/T118527#1814779 (EBernhardson) The oldest file i see is CirrusSearchRequests.log-20150726.gz. A 90 day retention...
[20:46:26] <wikibugs>	 Analytics, CirrusSearch, Discovery, operations, and 2 others: Delete logs on stat1002 in /a/mw-log/archive that are more than 90 days old. - https://phabricator.wikimedia.org/T118527#1814896 (Ottomata) Ok, yup, bug!  the job that removed old files was using ctime instead of mtime, and apparently th...
[20:47:21] <wikibugs>	 Analytics-Kanban, CirrusSearch, Discovery, operations, and 2 others: Delete logs on stat1002 in /a/mw-log/archive that are more than 90 days old. - https://phabricator.wikimedia.org/T118527#1814901 (Ottomata)
[20:50:34] <wikibugs>	 Analytics-Kanban, CirrusSearch, Discovery, operations, and 2 others: Delete logs on stat1002 in /a/mw-log/archive that are more than 90 days old. - https://phabricator.wikimedia.org/T118527#1814908 (Ottomata) Let's wait a few days and make sure this is working, then I think we can close.
[20:53:07] <ebernhardson>	 odd thing i just saw in hive...querying the wmf.webrequest table some rows have uri_path of `http://en.wikipedia.org/w/api.php`. I suppose it could be bad clients but not sure
[20:53:25] <ebernhardson>	 (not urgent to fix, just a curiosity)
[20:53:40] <wikibugs>	 Analytics-Kanban: Backfill daily-top-articles in cassandra [2015-09-01 - 2015-11-16 (included)]  - https://phabricator.wikimedia.org/T118991#1814915 (JAllemandou) NEW
[20:55:35] <wikibugs>	 Analytics-Kanban: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article - https://phabricator.wikimedia.org/T118933#1814938 (JAllemandou) Job restarted without spiders (or at least what we identify as Spiders). Restart Day : 2015-11-17 (testing day). Backfilling is tracked using https://phabric...
[20:56:23] <wikibugs>	 Analytics-Kanban: Backfill daily-top-articles in cassandra [2015-09-01 - 2015-11-16 (included)] {slug} - https://phabricator.wikimedia.org/T118991#1814942 (JAllemandou)
[20:56:36] <joal>	 hey milimetric
[20:57:30] <joal>	 ebernhardson: There are some oddities in wmf.webrequest indeed
[20:58:14] <joal>	 ebernhardson: I have not managed to figure out if it was coming from varnish-kafka, if data is actually really formatted in a wrong fashion, or if it's something else
[21:02:56] <wikibugs>	 Analytics-Backlog, Wikimedia-Developer-Summit-2016: Developer summit session:  Pageview API  overview - https://phabricator.wikimedia.org/T112956#1814979 (GLCiampaglia) Along the same lines of what @HYanWong proposed, it would be nice if it could be possible to specify a list of articles and the API could...
[21:10:42] <ottomata>	 nuria:
[21:10:42] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/254030/
[21:10:42] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/254031/
[21:10:47] <ottomata>	 +1s please :)
[21:10:49] <nuria>	 yess
[21:11:32] <nuria>	 ottomata: thank you!
[21:14:28] <wikibugs>	 Analytics-Backlog: Wikimedia "top" pageviews API has problematic double-encoded JSON - https://phabricator.wikimedia.org/T118931#1815062 (Milimetric) >> 2. Is the current format OK if we just decode it and present it uniformly as JSON? >  > I don't understand why the data was (poorly) double-encoded in the fi...
[21:21:14] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1815102 (Milimetric) I forgot to follow up on this, btw.  There's another restriction I found in the EL mysq...
[21:22:55] <ottomata>	 nuria, docs updated too, fyi
[21:23:03] <nuria>	 ottomata: many thanks sir
[21:32:26] <wikibugs>	 Analytics-Backlog, MediaWiki-extensions-WikimediaEvents, The-Wikipedia-Library, Wikimedia-General-or-Unknown: Implement Schema:ExternalLinkChange - https://phabricator.wikimedia.org/T115119#1815138 (Sadads) >>! In T115119#1814343, @Halfak wrote: > I'm not sure that truncation is an effective strat...
[21:39:49] <nuria>	 madhuvishy: yt? one question...
[21:39:55] <madhuvishy>	 nuria: yeah
[21:39:59] <madhuvishy>	 tell me
[21:40:24] <nuria>	 madhuvishy: when we run camus last, we did not need anything beyond properties file to make it write data into hdfs , right?
[21:40:53] <madhuvishy>	 ummm, we had some jars
[21:41:02] <madhuvishy>	 but that's it
[21:41:22] <madhuvishy>	 nuria: ^
[21:41:32] <nuria>	 madhuvishy: k
[21:41:45] <nuria>	 madhuvishy: let me try my 1st test and i will let you know
[21:41:57] <madhuvishy>	 nuria: ok cool, let me know if you need any help
[22:24:40] <milimetric>	 madhuvishy: mforns: I'm heading to dinner in a few minutes
[22:24:51] <milimetric>	 lemme know if you need anything
[22:25:08] <milimetric>	 (not from the dumpling place I'm going to)
[22:25:18] <madhuvishy>	 :)
[22:25:40] <madhuvishy>	 I think I'm good, running into some weird error but i can figure it out :)
[22:25:48] <madhuvishy>	 have fun at dinner
[22:27:52] <milimetric>	 thx, have a nice night
[22:34:01] <HaeB>	 nuria: (not urgent, but:) just read https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hive/Troubleshooting and it says this:
[22:34:06] <HaeB>	 "If your job is finished, you can find all of the job logs in HDFS at: /var/log/hadoop-yarn/apps/USER/logs/application_id/*"
[22:34:36] <HaeB>	 ..but /var/log/hadoop-yarn/ is empty
[22:34:40] <mforns>	 milimetric, ok :]
[22:34:46] <mforns>	 see you tomorrow!
[22:41:34] <nuria>	 HaeB: ya, it might be old info, let me see about the new path
[22:42:41] <nuria>	 HaeB: i think things are here: /mnt/hdfs/var/log/hadoop-yarn/apps/
[22:42:57] <nuria>	 HaeB: if you can update wiki it will be great
[22:44:05] <nuria>	 HaeB: wikitech doesn't load for me
[22:45:23] <nuria>	 HaeB: nevermind, just updated it
[22:45:50] <HaeB>	 nuria: cool, thanks!
[22:47:16] <nuria>	 madhuvishy: I think i am running into issues due to the checks that joal added recently
[22:47:21] <nuria>	 https://www.irccloud.com/pastebin/e1ojwJn0/
[22:50:19] <joal>	 Doesn't seem related nuria :)
[22:51:05] <joal>	 camus and checker are two distinct program, and the log lines you are showing seem to come from camus
[22:51:07] <joal>	 nuria: --^
[22:51:39] <nuria>	 joal: man it is kind of late, do not worry, i think i need to create a fake offset file
[22:51:45] <nuria>	 on tmp/nuria/history/2015-10-05-18-32-57/offsets-previous
[22:52:17] <joal>	 depends on what you are trying :)
[22:54:23] <nuria>	 joal: we can talk about it tomorrow, no worries
[22:54:31] <joal>	 k muria :)
[22:54:35] <joal>	 nuria sorry
[23:14:00] <mforns>	 madhuvishy, yt?
[23:14:15] <madhuvishy>	 mforns: hey yes
[23:14:44] <mforns>	 hey, do you get errors when executing wikimetrics tests?
[23:14:49] <mforns>	 in master I mean :]
[23:15:46] <madhuvishy>	 mforns: oh, I never tried
[23:15:48] <madhuvishy>	 let me
[23:16:03] <mforns>	 ok thanks
[23:16:45] <mforns>	 I get 3 errors, and when I try it with my code, the tests I added pass, but another test fails, the thing is, the code I wrote is totally independent from the rest of the tests...
[23:24:26] <madhuvishy>	 mforns: sorry i recently destoryed my vagrant, so it seems to be installing some dependencies from scratch
[23:24:45] <mforns>	 madhuvishy, don't worry hehe...vagrant
[23:26:02] <grrrit-wm>	 (PS1) Nuria: [WIP] Adding test schema to test json->avro publishing [analytics/refinery/source] - https://gerrit.wikimedia.org/r/254066
[23:26:36] <wikibugs>	 Analytics-Backlog, Team-Practices, User-JAufrecht: Get regular traffic reports on TPG pages - https://phabricator.wikimedia.org/T99815#1815469 (JAufrecht)
[23:28:55] <mforns>	 madhuvishy, I will push my change anyway with WIP so that we can sync both changes. There's a detail about the format in the results we can talk tomorrow
[23:29:09] <madhuvishy>	 mforns: okay cool
[23:29:38] <grrrit-wm>	 (PS1) Mforns: Add aggregate by user report [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/254068 (https://phabricator.wikimedia.org/T117287)
[23:30:18] <grrrit-wm>	 (CR) Mforns: [C: -1] "Still WIP" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/254068 (https://phabricator.wikimedia.org/T117287) (owner: Mforns)
[23:30:44] <mforns>	 madhuvishy, here is it ^
[23:30:49] <mforns>	 a-team, see you tomorrow!
[23:32:32] <madhuvishy>	 good night mforns
[23:32:43] <madhuvishy>	 btw on the tests i get something like
[23:32:47] <madhuvishy>	 https://www.irccloud.com/pastebin/aDia4zTO/
[23:32:48] <mforns>	 aha
[23:33:03] <madhuvishy>	 dont know what's this nosetests thing
[23:33:11] <madhuvishy>	 but we can talk about it tomorrow
[23:33:16] <mforns>	 ok
[23:33:28] <mforns>	 you seem to get the same 4 failures
[23:33:40] <mforns>	 ok, we talk tomorrow, thanks!
[23:33:44] <madhuvishy>	 ok :)
[23:33:50] <mforns>	 bye :]
[23:33:55] <madhuvishy>	 cya
[23:54:28] <bd808>	 nuria: would you or somebody on your team have an hour tomorrow or Friday to talk me through the next steps for my api analytics process? I think I have the major parts outlined at https://www.mediawiki.org/wiki/User:BDavis_%28WMF%29/Projects/Action_API_request_analytics#Data_acquisition but I'm not sure of where to put all the pieces and haven't managed to find a good template project to follow from looking at wikitech and phabricator.
[23:54:55] <bd808>	 I'd be more than happy to start such a top to bottom tutorial as part of paying back for the hand holding
[23:57:18] <wikibugs>	 Analytics-Backlog, Fundraising research, Research-and-Data: FR tech hadoop onboarding - https://phabricator.wikimedia.org/T118613#1815565 (atgo) Sure - the list of fr-tech engineers is: agreen, awight, cdentinger, dkozlowski, eeggleston. I'd also love any additional tutorials that are available.  Who i...