[00:09:06] mforns: the incoming stream of events, you can look at client-side-log in /var/log/eventlogging in 1002 or vanadium (1002 will do) [00:10:34] mforns: or beta labs, we can look at this tomorrow [00:10:46] ok nuria [00:11:05] I posted a comment on the task [00:14:58] nuria, what is the implication of the semicolon being/not being in the raw logs? [00:19:13] mforns: if events are not "cut" they have a semicolon at the end, it is used a delimiteter(?) [00:19:26] * delimeter [00:19:46] well the ones I reproduced have the semicolon at the end [00:21:05] no wait! some of them do NOT have it [00:21:22] mforns: ok, one less thing to worry about. Let's please test this on beta labs: https://wikitech.wikimedia.org/wiki/EventLogging/Testing/BetaLabs [00:22:13] ok, I'll do that tomorrow [00:25:37] mforns: Thank you! [00:28:14] nuria, de nada :] [00:28:49] Analytics-Tech-community-metrics, MediaWiki-Developer-Summit-2015, ECT-March-2015: Achievements, lessons learned, and data related with the MediaWiki Developer Summit 2015 - https://phabricator.wikimedia.org/T87514#1085157 (Rfarrand) Feedback form no longer accepting responses. Will begin looking at t... [09:34:49] (PS2) Joal: Add mobile monthly uniques job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 [11:25:32] Analytics, Labs, Tool-Labs: Make anonymized clickstream data available to the public - https://phabricator.wikimedia.org/T91495#1087859 (yuvipanda) NEW [12:40:27] Analytics-Tech-community-metrics, Possible-Tech-Projects, ECT-March-2015, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1088193 (Sai_Kiran) Hello I would like to work on this project for GSoC 2015. I have a pretty good idea... [12:53:27] Analytics-Tech-community-metrics, Possible-Tech-Projects, ECT-March-2015, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1088200 (NiharikaKohli) @Sai_Kiran, you could do the microtask in a Github repo and then ping the listed... [12:53:45] Analytics-Tech-community-metrics, Possible-Tech-Projects, ECT-March-2015, Epic: Allow contributors to update their own details in tech metrics directly - https://phabricator.wikimedia.org/T60585#1088201 (Qgil) @Dicortazar, I wonder whether such microtask is a bit too ambitious. Is this something th... [15:06:33] Analytics, MediaWiki-extensions-Gadgets, Possible-Tech-Projects: Gadget usage statistics - https://phabricator.wikimedia.org/T21288#1088440 (NiharikaKohli) @yuvipanda, willing to mentor this for the upcoming GSoC/Outreachy round? Is this big enough for a 3-month project? [15:07:33] Analytics, MediaWiki-extensions-Gadgets, Possible-Tech-Projects: Gadget usage statistics - https://phabricator.wikimedia.org/T21288#1088455 (yuvipanda) @NiharikaKohli sadly even starting on this requires a signed NDA with the WMF and access to the stat* boxes, so I'd say this is out of reach for GSoC... [15:08:24] Analytics, MediaWiki-extensions-Gadgets, Possible-Tech-Projects: Gadget usage statistics - https://phabricator.wikimedia.org/T21288#1088464 (NiharikaKohli) @yuvipanda, right. Thanks for clarifying. [15:13:52] (CR) Ottomata: [C: 2 V: 2] Include search consistently [analytics/refinery/source] - https://gerrit.wikimedia.org/r/193985 (owner: OliverKeyes) [15:43:59] yo nuria! [15:44:01] 2 thangs: [15:44:25] 1. can you review joseph's hql stuff for monthly mobile uniques (not a huge hurry atm), but a lot of hte sql is new to me [15:44:31] 2. can I help with the wikimetrics thing? [15:44:34] oh! [15:44:40] it is your bday! you are not working. [15:44:42] HAPPY BDAY! [15:44:43] NM [15:49:59] (CR) Ottomata: Add mobile monthly uniques job in oozie. (5 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 (owner: Joal) [16:52:51] Analytics, MediaWiki-extensions-MultimediaViewer, Multimedia, Multimedia-Sprint-2015-03-04, Patch-For-Review: Set up varnish 204 beacon endpoint for virtual media views and use it in Media Viewer - https://phabricator.wikimedia.org/T89088#1089033 (Gilles) [17:04:44] Analytics, Mobile-Apps, Scrum-of-Scrums, Wikipedia-App-Android-App, and 4 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1089065 (dr0ptp4kt) @BBlack, okay if we model after https://gerrit.wikimedia.org/r/#/c/120617/ ? [17:13:24] (Restored) Ottomata: (WIP) project class/variant extraction UDF [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188588 (owner: OliverKeyes) [17:13:59] Ironholds: hi! q for you [17:14:01] if you are around [17:22:18] Analytics, MediaWiki-extensions-Gadgets, Possible-Tech-Projects: Gadget usage statistics - https://phabricator.wikimedia.org/T21288#1089112 (Se4598) @yuvipanda: I don't know which infos are all on stat*, but relative easy queries can already be made together with [[https://www.mediawiki.org/wiki/Exte... [17:30:12] ottomata, hi, I have a problem, can not ssh into deployment-eventlogging02.eqiad.wmflabs [17:30:29] I suppose that I'm not part of its project [17:31:12] hello! [17:31:13] uhhh [17:31:20] oh yes, you need to be part of deployment project [17:31:23] you have to ask those folks [17:31:27] bd808: maybe can help? [17:31:29] I think I'm only in bastion and analytics [17:31:49] ok ottomata , so should I ask in the labs channel? [17:32:01] ok [17:32:04] thanks! [17:32:07] yup [17:41:34] ottomata, what's the Q? [17:41:39] does it involve NY food co-ops? ;p [17:45:35] HA! [17:45:39] man i always get you too mixed up [17:45:45] sometimes I send work emails to my other friend oliver [17:46:31] Ironholds: [17:46:47] i want to extract unique page title (with project domain?) from url fields [17:46:49] how? [17:46:50] :) [17:47:08] if I just grab uri_host + uri_path, i get things like [17:47:09] en.wikipedia.org/wiki/Main_Page 4290 [17:47:09] en.wikipedia.org/w/index.php 2606 [17:47:09] commons.wikimedia.org/w/index.php 1851 [17:47:09] zh.wikipedia.org/w/index.php 904 [17:47:09] de.wikipedia.org/w/index.php 657 [17:47:10] es.wikipedia.org/w/index.php 363 [17:47:10] fr.wikipedia.org/wiki/Hi%C3%A9rarchie 322 [17:47:11] es.wikipedia.org/wiki/Ferrocarril 320 [17:47:11] ru.wikipedia.org/w/index.php 308 [17:47:12] en.wikipedia.org/wiki/File:Squeeze_down_in_the_valley.jpg 288 [17:48:22] ottomata, you're screwed [17:48:41] haha [17:48:46] what we need is a function that takes uri_host, uri_path and uri_query [17:48:56] or, joined from mw table with page_id, eh? [17:48:58] if the uri_path is index.php it runs through a logic chain to extract particular parameters [17:49:07] then it normalises the result [17:49:18] ottomata, yeah, except Apps isn't passing that pageid through [17:49:27] aye, i dont' care about apps (right now! :p) [17:49:34] i'm just getting prototype :) [17:49:48] then, sure. Throw page table up in squoop and use the XFF value extractor to get something you can INNER JOIN ? [17:49:49] sqooping and joining sounds complicated though, HMMm [17:50:06] i'm not doing this in hive, it is totally possible though [17:50:10] ohh [17:50:13] i think i would rather just parse url for now [17:50:13] what are you doing it in? [17:50:16] spark streaming [17:50:19] aha [17:50:22] realtime trending reads over a windowed period :) [17:50:34] well, parsing URL will be harder because API, but it's workable [17:50:40] and it means you can include apps(!!) if you choose [17:50:46] anything messy is fine [17:50:51] i just want something that looks right [17:50:55] it doesn't have to be right :) [17:51:09] so,i should just make a big regex somehow? [17:51:19] to extract title=__ if it exists? [17:51:26] otherwise just use uri_path? [17:52:09] Analytics-Cluster, Analytics-Kanban: Update documentation page for the refined webrequest table in hive - https://phabricator.wikimedia.org/T90726#1089360 (JAllemandou) Open>Resolved [17:52:38] oh, that's cute [17:52:40] * Ironholds pats ottomata [17:53:30] cute is exactly what I want! [17:53:31] hha [17:53:38] okay, you want something that looks like this: [17:54:12] is the uri_path index.php or blank? if not, use uri_path [17:54:58] if it is index.php or blank, does the uri_query include edit? If so, discount [17:55:06] extract title, go forward [17:55:12] before I get to extracting [17:55:16] i a using your pageview def to filter [17:55:18] obviously you'll only be hitting the app API views if you're using the PV def [17:55:19] snap [17:55:28] that makes things easier; those all have a title= field. [17:55:34] they all do? [17:55:36] oh api ones? [17:55:38] * Ironholds thinks [17:55:44] I mean, the app ones should [17:56:19] oh, I tell a lie [17:56:27] they use action=mobileview and then page=foobarbaz [17:56:31] not title=foobarbaz [17:56:37] if blank, extract title? [17:56:41] because consistency is for chumps [17:56:43] i can get to a page if uri_path is blank? [17:57:13] ottomata, [17:57:14] https://en.wikipedia.org/?title=Barack_Obama [17:57:16] god hates us [17:57:20] ha, ok [17:58:08] if uri_path is index.php or blank { [17:58:08] if uri_query has title, return title [17:58:08] if uri_query has page, return page [17:58:08] } [17:58:08] else { [17:58:09] return uri_path [17:58:09] } [17:58:28] good enough? [17:58:58] extracting title via regex will be annoying...unless i figure out how to make scala regexes non greedy i guess. hm [17:59:05] maye I will find a scala url parser [17:59:06] yes ok. [18:01:28] ottomata, actually, you don't need a regex [18:01:43] take a look at the Java in the XFF value extractor in refinery-source [18:01:54] hmm, ok will do. [18:02:01] thanks Ironholds [18:02:02] lunchtime [18:02:42] np! [18:10:08] (CR) Ewulczyn: "I'm not sure when I will get to this (definitely not until March 25th), but I can do it." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/188588 (owner: OliverKeyes) [18:27:44] Analytics: Add CORS headers to datasets.wikimedia.org - https://phabricator.wikimedia.org/T91532#1089439 (DarTar) NEW [18:28:46] Analytics: Configure CORS on datasets.wikimedia.org - https://phabricator.wikimedia.org/T91532#1089446 (DarTar) [18:44:31] (PS3) Joal: Add mobile_app_uniques_monthly job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 [18:53:32] Analytics, Wikimedia-Fundraising: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1089515 (awight) @kevinator, I hear you might be interested in this problem. We need to do something like, digest web requests into a database that can be queried effi... [19:15:22] Analytics, Language-Engineering, MediaWiki-extensions-UniversalLanguageSelector, Mobile-Apps, and 4 others: there should be a comparison of clicks count on interlanguage on different platforms - https://phabricator.wikimedia.org/T78351#1089563 (DarTar) If this doesn't require new instrumentation I... [19:17:07] Analytics-Tech-community-metrics, Phabricator, Wikimedia-Hackathon-2015, ECT-March-2015: Metrics for Maniphest - https://phabricator.wikimedia.org/T28#1089574 (Aklapper) Adding the "Wikimedia-Hackathon-2015" project here. Even though I'm going to work on this before, there's larger potential to dis... [19:42:46] (CR) Ottomata: Add mobile_app_uniques_monthly job in oozie. (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 (owner: Joal) [20:26:59] (CR) Ottomata: "In IRC we decided to go wtih 'apps' instead of 'app' as the first patch on this had. Close to +2 :)" (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 (owner: Joal) [20:32:58] Analytics, Wikimedia-Fundraising: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1089982 (AndyRussG) I know nothing about whether it's easy or feasible... but I'm imagining something like a cluster-side script that logs certain requests directly in... [20:53:29] Analytics, operations: investigate txstatsd error logs - https://phabricator.wikimedia.org/T91464#1090061 (BBlack) [20:55:45] Analytics, operations: investigate txstatsd error logs - https://phabricator.wikimedia.org/T91464#1090071 (Ottomata) a:fgiunchedi>Ottomata [20:57:34] Analytics, Analytics-Kanban, Language-Engineering, Blocked-on-Analytics, LE-Sprint-83: Updated languages are not appearing on Language Dashboard - https://phabricator.wikimedia.org/T91369#1090088 (Milimetric) Open>Resolved a:Milimetric I had to manually update the TSV files where this da... [21:08:36] milimetric, yt? [21:22:05] Analytics, Fundraising Tech Backlog, Wikimedia-Fundraising, Fundraising Sprint Enya: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1090203 (Jalexander) Bump on this to see if any news :) [21:33:28] mforns: hi [21:33:35] hi [21:33:42] sorry reacting late, what's up [21:33:46] np [21:33:49] my head was in tag clouds :) [21:33:54] hehe [21:34:25] I was looking for that EventLogging diagram that Christian passed to me [21:34:33] with architecture of EL [21:34:46] I can not find it in Wikitech [21:34:53] do you have the link? [21:35:14] (looking) [21:35:32] (PS4) Joal: Add mobile_app_uniques_monthly job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 [21:37:53] mforns: :( sorry man, I can't find it yet [21:37:56] I know what you mean though [21:38:02] ok, no problem [21:38:03] the one he derived from reading puppet [21:38:41] I'm not sure of that, but it well may be [21:39:01] thanks anyway [21:43:04] Analytics, Fundraising Tech Backlog, Wikimedia-Fundraising, Fundraising Sprint Enya: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1090262 (awight) @Jalexander: February numbers should be ready in about half an hour, Can you access this URL? https://hue.wikimedia.org/bee... [21:51:28] (PS5) Joal: Add mobile_app_uniques_monthly job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 [21:52:32] Can anyone help me get the new password for EL data? [21:54:18] (PS6) Joal: Add mobile_app_uniques_monthly job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 [21:54:32] (PS7) Ottomata: Add mobile_apps_uniques_monthly job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 (owner: Joal) [21:54:44] (CR) Ottomata: [C: 2 V: 2] Add mobile_apps_uniques_monthly job in oozie. [analytics/refinery] - https://gerrit.wikimedia.org/r/194199 (owner: Joal) [22:10:54] Is there any way I can share a beeswax query with another user, e.g. https://hue.wikimedia.org/beeswax/execute/query/40 [22:12:01] (PS1) Joal: Refactor mobile_apps_uniques_daily to match newly baked monthly. [analytics/refinery] - https://gerrit.wikimedia.org/r/194400 [22:19:06] Analytics-Cluster, Analytics-Kanban, Easy: Mobile Apps PM has monthly report from oozie about apps uniques [8 pts] - https://phabricator.wikimedia.org/T88308#1090363 (JAllemandou) Open>Resolved [22:19:42] Analytics-Cluster, Analytics-Kanban, Easy: Mobile Apps PM has monthly report from oozie about apps uniques [8 pts] - https://phabricator.wikimedia.org/T88308#1008820 (JAllemandou) Code reviewd and merged, Andrew plans to deploy tomorrow, and when the jobs finish we should have some data :) [22:20:56] Analytics-Cluster, Analytics-Kanban: Refactor MobileApps uniques HQL to use external table to format data [8 pts] - https://phabricator.wikimedia.org/T90730#1090383 (JAllemandou) a:JAllemandou [22:22:50] Analytics, MediaWiki-Vagrant: role::hadoop will not provision on Ubuntu 14.04 (MediaWiki-Vagrant default) - https://phabricator.wikimedia.org/T70302#1090390 (JAllemandou) Tested that pretty badly, and I confirm : It works ! I think it can be closed. [22:45:08] Analytics, Fundraising Tech Backlog, Wikimedia-Fundraising, Fundraising Sprint Enya: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1090478 (awight) a:awight [22:46:31] Analytics, Fundraising Tech Backlog, Wikimedia-Fundraising, Fundraising Sprint Enya: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1090479 (Jalexander) >>! In T90635#1090262, @awight wrote: > @Jalexander: February numbers should be ready in about half an hour, > > Can you... [22:54:47] (CR) Mforns: [C: 2 V: 2] Analyze edit success rate by user type [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/192944 (https://phabricator.wikimedia.org/T89729) (owner: Milimetric) [23:11:00] Analytics-Kanban, Analytics-Visualization: Remove isZero field from data in Pentaho - https://phabricator.wikimedia.org/T91587#1090589 (kevinator) NEW [23:12:25] Analytics-Kanban, Analytics-Visualization: Remove isZero field from data in Pentaho - https://phabricator.wikimedia.org/T91587#1090596 (kevinator) [23:43:08] https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo [23:43:10] danke [23:45:50] https://meta.wikimedia.org/wiki/List_of_Wikipedias [23:46:00] Analytics, Fundraising Tech Backlog, Wikimedia-Fundraising, Fundraising Sprint Enya: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1090682 (awight)