[00:07:00] Analytics-Engineering, VisualEditor: VE-related data for the Galician Wikipedia - https://phabricator.wikimedia.org/T86944#1516826 (Jdforrester-WMF) [00:14:12] Analytics, MediaWiki-Authentication-and-authorization, Reading-Infrastructure-Team, MW-1.26-release, and 2 others: Create dashboard to track key authentication metrics before, during and after AuthManager rollout - https://phabricator.wikimedia.org/T91701#1516841 (Tgr) https://grafana.wikimedia.org... [09:39:50] Analytics, Labs, Labs-Infrastructure, Labs-Sprint-108, Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1517551 (akosiaris) I am gonna recap this a bit just to make sure I 've understood correctly. * People are a... [13:22:15] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [13:24:15] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [13:35:59] Analytics, Labs, Labs-Infrastructure, Labs-Sprint-108, Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1517912 (Ottomata) Gonna have to ping @kevinator and @dartar on that one. Note that people already have the... [13:41:50] Deskana|Away: in short, yes, Wikimetrics can serve an API request, but right now for privacy reasons you have to authenticate. Amanda Bittaker is working on a request for something like what you probably want. You should talk to her. [13:47:09] to answer ebernhardson's question: select from_unixtime(unix_timestamp(20150102030410)); [13:47:41] (but he's not here so I sent him an email too) [14:13:37] Analytics-EventLogging, Patch-For-Review: Kafka Client for MediaWiki - https://phabricator.wikimedia.org/T106256#1518061 (Ottomata) > Will this do anything to say, the consumability of data through Spark? Yes and no, It'll actually make things a little easier, especially if you are using Scala or Java. Y... [14:42:58] hi joal [14:43:05] Hi milimetric :) [14:43:11] i'm having confusions over the setup again [14:43:25] cave or irc ? [14:43:37] cave if you're available [14:43:41] I am ! [14:47:05] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [14:49:15] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [14:57:15] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [15:01:26] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [15:35:38] Analytics-Kanban: Date formatting bug on Vital Signs {crow} - https://phabricator.wikimedia.org/T108337#1518293 (Milimetric) NEW a:Milimetric [15:44:26] (PS1) Milimetric: Fix date formatting, assume UTC [analytics/dashiki] - https://gerrit.wikimedia.org/r/230117 (https://phabricator.wikimedia.org/T108337) [15:52:05] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [15:56:15] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [16:10:46] Analytics-Backlog, Analytics-EventLogging: Make EventLogging alerts based on Kafka metrics {stag} - https://phabricator.wikimedia.org/T106254#1518410 (ggellerman) p:High>Normal [16:10:47] Analytics-Backlog, Analytics-EventLogging: Change EventLogging Alerts to be more reliable {stag} - https://phabricator.wikimedia.org/T108339#1518411 (kevinator) NEW [16:11:18] Analytics-Backlog, Analytics-EventLogging: Change EventLogging Alerts to be more reliable {stag} - https://phabricator.wikimedia.org/T108339#1518424 (kevinator) [16:12:31] Analytics-Backlog, Analytics-EventLogging: Change EventLogging Alerts to be more reliable {stag} - https://phabricator.wikimedia.org/T108339#1518438 (JAllemandou) Related to data not showing up in graphite at same rate. Maybe making the alert check for past 5 minutes could at least solve of the issue ? [16:18:08] Analytics-Backlog: Provide the Wikimedia DE folks with Hive access/training {flea} - https://phabricator.wikimedia.org/T106042#1518455 (kevinator) p:Normal>High [16:21:46] Analytics-Backlog, Analytics-Cluster: Classification of Bing robots as spider traffic instead of user traffic {hawk} - https://phabricator.wikimedia.org/T106134#1518461 (kevinator) p:Normal>High [16:24:49] Analytics-Backlog: Write script to calculate total point value of cards marked as resolved in a regular 1wk or 2 wk window - https://phabricator.wikimedia.org/T108211#1518470 (kevinator) [16:24:50] Analytics-Backlog: Write script to track cycle time of tasked tickets - https://phabricator.wikimedia.org/T108209#1518471 (kevinator) [16:25:40] madhuvishy: Heya, are you there ? [16:25:53] ssh shrek [16:25:59] oope :) [16:26:09] hey joal yes, just leaving to office. will you be around in about an hour? [16:26:13] Analytics-Backlog: Write scripts to track cycle time of tasked tickets and velocity - https://phabricator.wikimedia.org/T108209#1518473 (kevinator) p:Triage>High [16:26:33] madhuvishy: was looking aftger bob code again, I'll find it in my logs: ) [16:26:36] Thanks ! [16:28:10] joal: https://github.com/cervisiarius/wikimedia/blob/master/navigation_trees/src/main/java/org/wikimedia/west1/traces/GroupAndFilterMapper.java [16:28:23] madhuvishy: Thx a million :) [16:28:31] You've been faster than me ;) [16:29:01] Analytics-Backlog: Add better regexp to agent_type bot filtering - https://phabricator.wikimedia.org/T108343#1518531 (JAllemandou) NEW [16:33:34] Guys, I just scanned it rapidly, but seems very cool : http://radar.oreilly.com/2015/08/the-world-beyond-batch-streaming-101.html [16:34:09] Analytics-Backlog, Reading-Admin, Research-and-Data, Research consulting: Request for data: sites traffic by topics/ subject areas and geographies - https://phabricator.wikimedia.org/T107613#1518562 (Milimetric) [16:34:16] Analytics-Backlog, Team-Practices-This-Week: Get regular traffic reports on TPG pages - https://phabricator.wikimedia.org/T99815#1518563 (Milimetric) [16:34:25] Analytics, Analytics-Backlog, Research-and-Data, Research consulting: Too few page views for June/July 2015 - https://phabricator.wikimedia.org/T106034#1518566 (Milimetric) [16:34:31] Analytics-Backlog, Research-and-Data, Research consulting: Analysis on traffic through the HTTPS transition - https://phabricator.wikimedia.org/T102431#1518567 (Milimetric) [16:34:39] Analytics-Backlog, Research-and-Data, Research management: Pipeline from Research to productization - https://phabricator.wikimedia.org/T105815#1518568 (Milimetric) [16:36:42] Analytics-Backlog: Check and potentially timebox limn-language-data reports {tick} - https://phabricator.wikimedia.org/T107504#1518578 (Milimetric) [16:36:46] Analytics-Backlog: Check and potentially timebox limn-flow-data reports {tick} - https://phabricator.wikimedia.org/T107502#1518580 (Milimetric) [16:37:58] gotta grab some lunch, bbl [18:35:39] joal: sorry i had to run some last minute errands and just got to office [18:35:50] we can talk now if its not too late, or monday [19:43:46] Deskana: saw your question last night and responded, Kevin's going to talk about adding that API to wikimetrics next week [19:44:13] we can add your use case to that discussion and make it higher priority [19:44:21] milimetric: Yeah, thanks for the response! I actually asked the question because I was chatting to Amanda about it. :-) [19:44:28] oh, right :) [19:45:01] good then, I want to find more excuses to add that feature [19:45:13] What she's trying to do seems pretty simple, assuming there's a backend somewhere that you can shift the heavy lifting on to, such as Wikimetrics. [20:04:12] yeah, the API is already there, it's just behind OAuth because when we built it legal was concerned with leaking cohort membership information [20:35:46] Analytics-Cluster: Make varnishkafka produce using dynamic topics - https://phabricator.wikimedia.org/T108379#1519540 (Ottomata) NEW a:Ottomata [21:03:14] Analytics, Labs, Labs-Infrastructure, Labs-Sprint-108, Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1519641 (Halfak) Hey folks. I just wanted to hop in to +1. I put a new dataset up on datasets.wikimedia.org... [21:13:42] Analytics-Backlog, Research-and-Data, RD-2016Q1, Research management: Pipeline from Research to productization - https://phabricator.wikimedia.org/T105815#1519681 (DarTar) [21:27:43] Analytics, MediaWiki-Authentication-and-authorization, Reading-Infrastructure-Team: Kill MediaWiki.authmanager.login.*.failure. statsd buckets - https://phabricator.wikimedia.org/T108386#1519722 (Tgr) NEW a:Tgr [21:49:57] hi all, is there a tutorial on how to make a dashboard from hadoop data? [21:50:08] (is that a thing that's possible at all?) [21:56:39] milimetric: ^ [22:34:01] tgr, sure, it's possible but we have no scripts or schedulers in place to make it easy. In theory, you'd run some Hive in a cron on stat1002 and put the outputs in the /a/limn-public-data directory that gets rsynced and served via datasets.wikimedia.org [23:27:32] milimetric: so I'd have to do the legwork but from a resources/server load standpoint it would be fine? [23:27:50] I'm thinking about collecting API usage and latency data