[00:59:18] Analytics-Backlog, Research-and-Data, Research consulting: Scope effort needed to validate token-based Unique Client data - https://phabricator.wikimedia.org/T107239#1490380 (DarTar) NEW a:DarTar [01:12:58] milimetric: Hi [01:13:13] milimetric: I recall seeing a browser_name/version in some database recently in analytics context [01:13:19] as column [01:13:48] I presumed that all raw page view data was preprocessed with ua-parser before storing in there but that isn't the case then? [01:14:25] Krinkle, yeah, that's the case. The Hive tables that are overlayed on top of the HDFS data have those columns [01:14:53] Aha [01:15:06] the most useful table is probably the hourly aggregated wmf.pageview_hourly [01:15:31] milimetric: And that's unsampled, for all wiki domain html page views? [01:15:50] yes [01:17:40] Krinkle, you have access to that, right? [01:18:17] milimetric: I'd like to, but if that's unusual I'm fine with not having access. [01:18:45] I don't think I do, but let's see. Where do I go? I'm not familiar with how Hive/Hadoop actually work from a querying perspective. [01:18:51] or what program to invoke where [01:27:02] oops [01:27:22] Krinkle: ssh into stat1002.eqiad.wmnet [01:27:32] run "hive" [01:28:26] from there it's just like a mysql database with a few quirks. Try not to run over too much data and I'm totally fine with you running a reasonable amount of queries there [01:29:19] but in the meantime hopefully we'll get to work on the new traffic reports based on this data. We have the pageview API and event logging prioritized above that [01:52:47] milimetric: auth failure for ssh user 'krinkle' [01:53:07] But yeah, I'll keep it basic. [01:53:16] Would love to query this a bit from time to time [01:56:02] milimetric: Is https://wikitech.wikimedia.org/wiki/Analytics/Data_access up to date? If so, which group should I ask for? [01:56:07] analytics-users? [02:09:16] Krinkle you need statistics-privatedata-users and yes that page is up to date [02:09:32] milimetric: Aside from the thing about RT tickets ;-) [02:09:39] Right :) [02:13:25] * Krinkle updates [02:21:04] milimetric: Thanks, https://phabricator.wikimedia.org/T107243 [02:21:12] https://wikitech.wikimedia.org/w/index.php?title=Analytics%2FData_access&type=revision&diff=172193&oldid=153389 [04:43:45] Analytics-Wikistats: Remove links in wikistats to minnan.wikipedia.org - https://phabricator.wikimedia.org/T107250#1490654 (Glaisher) NEW [06:19:26] Analytics-Kanban: Event Logging sends mysql consumer stats to statsd [5 pts] {oryx} - https://phabricator.wikimedia.org/T105935#1490759 (madhuvishy) [07:14:08] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1490798 (jgbarah) NEW [07:26:56] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1490814 (jgbarah) [07:29:05] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1490817 (Qgil) p:Triage>Low [07:32:44] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1490821 (Qgil) I hadn't notice that there is no closing event in our migrated tickets. I don't think we are going to touch that now, so I guess we will have to live with se... [07:37:32] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [07:39:32] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [08:17:48] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1490850 (jgbarah) OK, waiting for @chasemp comments, if any. [08:37:46] Analytics-Wikistats: statistics for Wikidata API usage - https://phabricator.wikimedia.org/T64873#1490866 (Addshore) Actually I guess https://gerrit.wikimedia.org/r/#/c/226640/ is related here [09:10:40] Analytics-Tech-community-metrics, ECT-August-2015: Fine tune "Code Review overview" metrics page in Korma - https://phabricator.wikimedia.org/T97118#1490949 (Qgil) This task was part of the June sprint but then slipped off the July sprint. Proposing it for August. [13:24:10] (PS2) Hashar: Ignore swp files [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/187065 (owner: Milimetric) [13:25:21] (CR) Hashar: [C: 2 V: 2] Ignore swp files [analytics/mediawiki-storage] - https://gerrit.wikimedia.org/r/187065 (owner: Milimetric) [14:41:30] Analytics-Kanban: Clean up mobile-reportcard dashboards {frog} [13 pts] - https://phabricator.wikimedia.org/T104379#1491667 (kevinator) [14:41:31] Analytics, Analytics-Kanban, Patch-For-Review: Reportupdater: put history and pid files inside the project folder [5 pts] {lamb} - https://phabricator.wikimedia.org/T103385#1491666 (kevinator) Open>Resolved [14:42:03] Analytics-Kanban, Wikimania-Hackathon-2015: Dockerize Hadoop Cluster, Druid, and Samza + Load Test - https://phabricator.wikimedia.org/T102980#1491670 (kevinator) Open>Resolved [14:57:54] Analytics-Kanban: Clean up mobile-reportcard dashboards {frog} [13 pts] - https://phabricator.wikimedia.org/T104379#1491696 (mforns) http://datasets.wikimedia.org/limn-public-data/mobile/datafiles/page-ui-daily.csv seems to be receiving events normally, but mid February MobileWebUIClickTracking saw a big drop... [15:37:56] Analytics-EventLogging, Analytics-Kanban: Event Logging doesn't seem to handle unicode strings {oryx} [8 pts] - https://phabricator.wikimedia.org/T99572#1491803 (Milimetric) [15:39:20] Analytics-Cluster, Analytics-Kanban: Test Cassandra as a storage strategy {slug} [5 pts] - https://phabricator.wikimedia.org/T101786#1491813 (Milimetric) a:JAllemandou [15:39:30] Analytics-EventLogging, Analytics-Kanban: Event Logging doesn't seem to handle unicode strings {oryx} [8 pts] - https://phabricator.wikimedia.org/T99572#1491814 (Milimetric) a:Milimetric [16:21:33] analyticsy people, ottomata particularly: [16:21:45] if I search for a key in the x_analytics_map and it's not there, will it return NULL or an empty string? [16:24:21] great question! i do not know [16:24:35] experiment to find out! [16:24:43] lunchtime! [16:37:23] shall do! [17:53:36] milimetric, sorry, I was a little late to Scrum of Scrums. Analytics wants to know which reports at https://stats.wikimedia.org/ people are using? Any other sites affected? [17:54:03] matt_flaschen: this is a longer term question, I'll mention it again for sure [17:54:22] basically, wikistats in its current incarnation will no longer be supported around the end of this year [17:54:32] so we have to replace the reports that are of highest value [17:54:41] milimetric, okay, but we should start talking about it. And wikistats === stats.wikimedia.org, right? [17:54:50] I've started tracking this in a task here: https://phabricator.wikimedia.org/T107175 [17:55:02] yes, stats.wikimedia.org [17:55:46] milimetric, thanks, will give people a heads up and email our list (EE). [17:56:00] thanks matt_flaschen [19:00:27] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [19:02:37] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [19:17:31] ottomata, poke [19:17:42] just to check I'm not crazy; POST requests don't get their query strings logged in the request logs, right? [19:17:53] Ironholds: hi. [19:18:08] if there was a query string on a POST, then sure, why not? [19:18:21] POST http://boogers.boo?yes=no [19:18:22] sure [19:18:27] hmn [19:18:31] okay, interesting. Thanks! [19:18:54] For some reason I thought they didn't log [19:19:09] maybe there aren't any? [19:19:35] i guess that is unlikely, eh? there are probably lots of MW urls that always have query strings, even for post [19:20:51] wait, no, worked it out [19:20:53] more basic than that [19:21:02] uri_path RLIKE() OR uri_query RLIKE() [19:21:19] the sampled and unsampled logs having different formats is always confusing. Resolved, I think! We'll see what the query string says [19:21:23] thank you for being my rubber ottomata :D [19:21:26] yup! [19:34:21] Ironholds: Maybe you were thinking about the actual contents of the POST, which almost certainly isn't logged... I think it's rare to put any parameters in the URL when making a POST request. [19:35:14] awight|fud, oh there's actually a great example of this [19:35:21] users authenticate to our API using query parameters [19:35:27] with their normal wiki password [19:35:46] strictly-speaking I'm sitting on the passwords, in plaintext, of every user who has used AWB or similar automata in the last 3 months. [19:39:17] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL 20.00% of data above the critical threshold [30.0] [19:41:27] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK Less than 15.00% above the threshold [20.0] [19:47:50] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1492581 (chasemp) I'm confused on what is wanted here. Is the idea that tasks from bugzilla should have a kanban workboard style card movement transaction to close them out? [20:09:47] Analytics-EventLogging, Analytics-Kanban: Event Logging doesn't seem to handle unicode strings {oryx} [8 pts] - https://phabricator.wikimedia.org/T99572#1492631 (Milimetric) Steps taken to look for the problem: * looked through past 7 days of logs for "Invalid string" and found nothing * looked through g... [20:29:16] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1492680 (Qgil) No, this is about the blue lines flat between Jan 2002 and Nov 2014 at http://korma.wmflabs.org/browser/maniphest.html, because Bugzilla reports migrated to... [20:43:13] Analytics-Kanban: {lama} Wikistats 2.0 - https://phabricator.wikimedia.org/T107175#1492724 (Krinkle) [20:52:19] Analytics-Kanban: Clean up mobile-reportcard dashboards {frog} [13 pts] - https://phabricator.wikimedia.org/T104379#1492752 (mforns) http://datasets.wikimedia.org/limn-public-data/mobile/datafiles/watchlist-activity.csv is the same case as the latter. Moved it to reportupdater and it should work. Also shifted... [20:59:31] milimetric: around? [20:59:37] hey madhuvishy [21:00:42] I'm looking at the statsd thing. I found that the metrics are being written from here - https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/1a25864930f52bd19505012313d0d5a9bf6b9a2d/server/bin/eventlogging-reporter [21:01:01] right [21:01:07] and that runs on hafnium [21:01:36] but don't really know where the code to measure things on the consumer end should be [21:02:17] milimetric: ^ [21:02:20] right [21:02:33] i was thinking it'd just be in the mysql writer? [21:03:09] hereish: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/server/eventlogging/handlers.py#L259 [21:03:17] milimetric: ya that's what i thought - but making sure if it's a design thing to keep all reporting in the reporter file [21:03:41] yeah, that's nice and separate but there's no other real way of knowing when exactly the insert worked [21:03:49] this is somewhat related to something ottomata and I talked about [21:03:54] milimetric: right [21:04:02] which is the mysql writer should confirm when an event was handled properly [21:04:10] that way we could tell kafka we handled that message [21:04:14] aah [21:04:26] yeah for the auto commit stuff [21:04:31] to work properly [21:04:35] so if you want to take your time and think of a way to elegantly do this, it wouldn't be wasted effort [21:05:05] maybe like we have readers and writers we could have loggers [21:05:08] hey uhhhh, what are yall doing? [21:05:12] and one logger would be statsd [21:05:18] and another would be kafka auto-commit [21:05:23] eh... not sure :) [21:05:35] ottomata: i'm trying to work on https://phabricator.wikimedia.org/T105935 [21:05:38] my mind is on other elegance right now [21:05:39] eventlogging-reporter will be deprecated as part of stag [21:06:00] and wondering where the consumer statsd code should go [21:06:05] ottomata: ha [21:06:15] where would the metrics be reported from? [21:06:18] kafka [21:06:24] jm [21:06:25] jmx [21:06:29] via jmxtrans [21:06:31] hmmm [21:06:47] and, and think milimetric is right, if we did the consumers and offset commits right [21:06:59] the mysql consumer that is consuming from kafka topic eventlogging-all [21:07:03] hm, so maybe the right way to do this is to get the mysql writer to somehow talk back to kafka to auto-commit and then handle the reporting after that [21:07:04] would commit after it inserts. [21:07:13] so, you'd have a high water mark of which events you'd actually inserted into mysql [21:07:29] if you wanted a rate to look at in graphite, you could just track that [21:07:52] current high water mark - last high water mark / time period == inserts per second [21:08:03] there are tools that track this though [21:08:11] i think pykafka comes with some even... [21:08:59] https://github.com/Parsely/pykafka/blob/master/pykafka/cli/kafka_tools.py#L40 [21:09:24] or [21:09:39] http://quantifind.com/KafkaOffsetMonitor/ [21:10:31] ottomata: hmmm I was doing this using - ./kafka-run-class kafka.tools.ConsumerOffsetChecker --zookeeper localhost:2181 --group eventlogging-group [21:10:58] or https://github.com/linkedin/Burrow [21:11:02] ah cool! [21:11:03] yea! [21:11:46] oo coool [21:11:46] https://github.com/linkedin/Burrow/wiki/HTTP-Endpoint [21:11:55] https://github.com/linkedin/Burrow/wiki/http-request-consumer-group-status [21:12:19] that is awesome! [21:12:22] we should use that ! [21:12:47] anyway, ja, madhu, if you really want to send to statsd directly from the mysql consumer, you should do it there [21:12:52] in the consumer code [21:13:08] maybe as a aconfigurable option [21:15:14] ottomata: okay, i'll try to do that for now. [21:15:35] where do the stats go? I don't see eventlogging in graphite.wmflabs.org [21:15:41] is there a different instance? [21:16:01] dunno about wmflabs [21:16:08] it is here though [21:16:08] https://graphite.wikimedia.org/ [21:16:15] ah okay that would make sense [21:16:42] ottomata: is it normal it asks me for authentication? [21:17:22] madhuvishy: yeah, it should still be your wikitech credentials I think [21:17:50] what I look at there usually is eventlogging -> overall -> raw and eventlogging -> overall -> valid [21:17:58] what we'd want is eventlogging -> overall -> inserted [21:20:18] ottomata: hmmm cannot seem to login [21:21:57] hm, madhuvishy, according to wikitech i think you should be able to [21:21:58] https://wikitech.wikimedia.org/wiki/Graphite.wikimedia.org [21:22:23] Analytics-Kanban: {lama} Wikistats 2.0 - https://phabricator.wikimedia.org/T107175#1492880 (Neil_P._Quinn_WMF) @Milimetric, is this task mainly about the generation of raw statistics (that is, the kind stored at http://datasets.wikimedia.org), or about a consolidated dashboard for useful metrics, or both? [21:24:27] ottomata: :| [21:28:44] madhuvishy: look in grafana until we figure this out? [21:28:47] that's not restricte ( i think) [21:28:56] ottomata: okay [21:29:08] http://grafana.wikimedia.org/#/dashboard/db/eventlogging [21:29:18] it isn't as easy to browse with grafana [21:29:40] Analytics-Kanban: {lama} Wikistats 2.0 - https://phabricator.wikimedia.org/T107175#1492903 (Milimetric) @Neil_P._Quinn_WMF, and others: This task is a project epic task that we use as a place-holder to both remember what {lama} means and keep notes about the project. So discussions about all things related... [21:29:43] but, try editing one of those graphs ( don't save), and just changing metrics, or adding a query [22:09:49] milimetric: did we have an idea of the MySQL consumer bottleneck? [22:10:30] it looked like it inserted over 2k per second [22:10:39] but we didn't do as extensive a job testing that [22:12:31] https://wikitech.wikimedia.org/wiki/EventLogging/Performance#Event_Logging_load_test [22:20:46] Did the EEVS URL change? https://metrics.wmflabs.org/static/public/dash/ doesn't work any more (and hasn't for a few days). [22:22:13] James_F: yes, that's an outdated URL [22:22:20] it's now much more sensibly: https://vital-signs.wmflabs.org/ [22:22:36] however, we did remove most of the bad metrics [22:22:44] the editing metrics only updated like once in a blue moon [22:22:50] but the charts made them look otherwise [22:22:55] so we just took them down [22:23:01] Oh. [22:23:10] basically, labsdb couldn't support the queries [22:23:21] Right. [22:23:32] what were you looking for James_F [22:23:34] https://www.mediawiki.org/wiki/Analytics/Archive/Editor_Engagement_Vital_Signs etc. still list the old URL; possible to put in a placeholder link? [22:23:53] All the new user survival stuff. [22:24:04] ah yes, one sec [22:24:32] i'll update that page but give me a minute I'm finishing something up, I can enable the metrics again [22:24:42] i guess it's useful even if it's unreliably calculated [22:24:45] Yeah. :-) [22:32:32] James_F: https://vital-signs.wmflabs.org/#projects=ruwiki,itwiki,dewiki,frwiki,enwiki,eswiki,jawiki/metrics=RollingNewActiveEditor [22:33:53] milimetric: Awesome, thanks. [22:34:25] the other three are there. Notice that the new dashboard shows blanks on the dates where computation couldn't take place [22:37:37] Yeah, that's quite helpful (if only to highlight that we probably need to throw hardware at the problem). [22:39:24] it's not really hardware we need, the way labsdb mirrors the data is just not smart. You can get 100x the performance with just a few indices. And I measured 1000x improvement if we simply change the schema we replicate into. [22:40:35] I'm not a math wiz but that seems good [22:43:20] Analytics-Kanban: Clean up mobile-reportcard dashboards {frog} [13 pts] - https://phabricator.wikimedia.org/T104379#1493191 (mforns) http://datasets.wikimedia.org/limn-public-data/mobile/datafiles/diff-activity.csv sames as the last ones. Should now work. [22:50:09] Analytics-Tech-community-metrics: Closed tickets in Bugzilla migrated without closing event? - https://phabricator.wikimedia.org/T107254#1493228 (jayvdb) This is probably the main reason I still frequent the old bugzilla data. [23:04:01] Analytics-Kanban: Clean up mobile-reportcard dashboards {frog} [13 pts] - https://phabricator.wikimedia.org/T104379#1493275 (mforns) http://datasets.wikimedia.org/limn-public-data/mobile/datafiles/mobile-options.csv seems to be working fine in reportupdater. I created a non-timeboxed version of it called mobi... [23:09:54] milimetric: where would i test the mysql consumer? [23:17:25] madhuvishy, I think you can test it in the beta cluster [23:17:59] madhuvishy, deployment-eventlogging02.eqiad.wmflabs, no? [23:24:12] mforns: aah thanks. What are you doing up at this time! [23:24:33] :] going to sleep now [23:25:35] mforns: okay, good night :) [23:25:46] madhuvishy, deployment-eventlogging02.eqiad.wmflabs is theoretically a copy of eventlogging system, so you can deploy there and run your code [23:25:57] alright. [23:26:00] i'll test it there [23:26:03] some events arrive there, I don't know why, maybe teams testing their schemas [23:26:08] right [23:26:27] but you can use the load test script that is in the test folder [23:27:01] it points by default to the beta cluster [23:31:14] mforns: right, okay I'll look at it, thanks [23:31:30] madhuvishy, ok, good night! see you! [23:31:35] good night :) [23:40:18] (PS1) Mforns: Clean up mobile-reportcard reports and dashboards [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/227911 (https://phabricator.wikimedia.org/T104379) [23:41:39] (CR) Mforns: [C: -1] "Still WIP." [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/227911 (https://phabricator.wikimedia.org/T104379) (owner: Mforns) [23:43:44] (CR) Mforns: Clean up mobile-reportcard reports and dashboards (1 comment) [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/227911 (https://phabricator.wikimedia.org/T104379) (owner: Mforns)