[00:33:08] Analytics, Fundraising Sprint Enya, Fundraising Tech Backlog, Wikimedia-Fundraising: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1064504 (ellery) @awight I suspect its restricting to webrequest_source = 'text'. Maybe SRI goes into a different partition now. I fired off... [00:36:22] Analytics, Mobile-Apps, Wikipedia-App-Android-App, Wikipedia-App-iOS-App, operations: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1064522 (dr0ptp4kt) p:Triage>Normal [00:43:04] kevinator: what’s the preferred link to the Unique Clients (I’m updating the R&D goals page) [00:43:14] UC proposal, that is [00:44:43] you mean this: https://wikitech.wikimedia.org/wiki/Analytics/Unique_clients/Last_visit_solution [00:46:08] kevinator, do you know Joseph? can't find him in https://office.wikimedia.org/wiki/Contact_list [00:46:10] kevinator: I guess so, the parent page doesn’t exist so I’ll link to it for now [00:47:12] yurik: yes, Joseph Allemandou. He’s new so he has not been added to the staff web pages yet [00:47:32] yurik: he also goes by joal in IRC [00:47:46] and he’s in France [00:48:18] i thought we add ourselves... like ... in a wiki )) [00:48:56] ;-) [00:52:50] kevinator: I updated the goals page on mw.org per QR, FYI https://www.mediawiki.org/wiki/Wikimedia_Engineering/2014-15_Goals#Research_and_Data [04:37:27] Analytics, Multimedia: Video view counts? - https://phabricator.wikimedia.org/T88775#1020132 (Tbayer) As @Tgr points out (and @ezachte confirmed on the talk page there), this proposal is related: https://www.mediawiki.org/wiki/Requests_for_comment/Media_file_request_counts [06:42:52] Analytics-Cluster, Analytics-Kanban, Easy: Mobile Apps PM has monthly report from oozie about apps uniques - https://phabricator.wikimedia.org/T88308#1064910 (Deskana) Great! Thank you! [09:40:41] (PS1) Gilles: Fix typo [analytics/multimedia] - https://gerrit.wikimedia.org/r/192774 (https://phabricator.wikimedia.org/T89814) [09:40:58] (CR) Gilles: [C: 2] Fix typo [analytics/multimedia] - https://gerrit.wikimedia.org/r/192774 (https://phabricator.wikimedia.org/T89814) (owner: Gilles) [09:41:04] (Merged) jenkins-bot: Fix typo [analytics/multimedia] - https://gerrit.wikimedia.org/r/192774 (https://phabricator.wikimedia.org/T89814) (owner: Gilles) [13:47:34] (PS1) Mforns: Merge branch 'master' into scheduler [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192795 [13:47:36] (CR) jenkins-bot: [V: -1] Merge branch 'master' into scheduler [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192795 (owner: Mforns) [14:53:46] Analytics-Cluster: Getting Ananth started - https://phabricator.wikimedia.org/T77196#1065762 (Ottomata) Open>declined a:Ottomata [15:11:45] Analytics: Could it be that the geo IP matching is not accurate for Africa? - https://phabricator.wikimedia.org/T90240#1065842 (ezachte) [15:16:13] Analytics-Cluster, Analytics-Kanban: WMF has technical documentation on UC by last visited date [5 pts] {bear} - https://phabricator.wikimedia.org/T88812#1065855 (kevinator) a:Nuria [15:27:48] (PS1) Ottomata: Fix bug where mobile_apps daily uniques archive files are named incorrectly [analytics/refinery] - https://gerrit.wikimedia.org/r/192807 [15:29:31] (PS2) Ottomata: Fix bug where mobile_apps daily uniques archive files are named incorrectly [analytics/refinery] - https://gerrit.wikimedia.org/r/192807 [15:34:14] (CR) Joal: [C: 1] "Good for me :)" [analytics/refinery] - https://gerrit.wikimedia.org/r/192807 (owner: Ottomata) [15:38:11] Analytics-Cluster, Analytics-Kanban: Add 'version' field to refined webrequest table in Hive - https://phabricator.wikimedia.org/T90725#1065912 (JAllemandou) NEW [15:40:17] Analytics-Cluster, Analytics-Kanban: Create a documentation page for the refined webrequest table in hive. - https://phabricator.wikimedia.org/T90726#1065940 (JAllemandou) NEW [15:41:39] joal: https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest [15:42:24] Thx ottomata, will use that as a starting point :) [15:42:44] (PS4) Mforns: [WIP] [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) [15:42:50] (CR) jenkins-bot: [V: -1] [WIP] [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [15:43:04] also [15:43:05] https://wikitech.wikimedia.org/wiki/Analytics/Data [15:43:33] Ok [15:44:02] thx ! [15:44:56] (PS5) Mforns: [WIP] [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) [15:45:02] (CR) jenkins-bot: [V: -1] [WIP] [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [15:46:09] Analytics-Cluster, Analytics-Kanban: Create a documentation page for the refined webrequest table in hive. - https://phabricator.wikimedia.org/T90726#1065964 (JAllemandou) Andrew created https://wikitech.wikimedia.org/wiki/Analytics/Data/Webrequest and already have content there. Update needed to follow c... [15:46:31] (PS3) Ottomata: Fix bug where mobile_apps daily uniques archive files are named incorrectly [analytics/refinery] - https://gerrit.wikimedia.org/r/192807 [15:46:38] (CR) Ottomata: [C: 2 V: 2] Fix bug where mobile_apps daily uniques archive files are named incorrectly [analytics/refinery] - https://gerrit.wikimedia.org/r/192807 (owner: Ottomata) [15:47:26] ottomata: let me know when you want us to deploy / change the webrequest table to add geo [15:47:51] (Abandoned) Mforns: Merge branch 'master' into scheduler [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192795 (owner: Mforns) [15:48:32] OOoooOOo [15:48:43] did you have another patch you were going to submit? i thought you said you did [15:48:46] if you didn't i'll merge that now too [15:49:27] Ohhh, just thought of another permission you need [15:49:39] Well, I wonder if we shouldn't put the refinery-hive jar version number as a parameter in the oozie job [15:49:49] Cause now, we need to change it ! [15:50:02] sure, we started doing that in a nother job [15:50:06] qchris and I already argued about this [15:50:08] and he won [15:50:09] so sure! [15:50:11] :) [15:50:12] or [15:50:14] milimetric, nuria: the gerrit link I pasted in the standup was wrong (I pushed a master merge by mistake), the correct change is this: https://gerrit.wikimedia.org/r/#/c/192319/ [15:50:17] in the hql file is fine [15:50:20] right? [15:50:23] hm [15:50:29] correct [15:50:37] oh, it is there [15:50:41] and set the parameter at bundle.properties level [15:50:42] you want to move it to oozie instead? [15:50:44] not a bad idea [15:50:45] ok [15:50:58] i can do that real fast, shall I? [15:51:08] I'll submit a patch, but give me some time to test first ;) [15:51:11] ok ok [15:51:13] go for it. [15:51:14] thanks mforns, I'll look in a bit [15:51:24] oh, also, we want to add version before we deploy this new refine job, right? [15:51:37] record_version? [15:51:40] table_version? [15:51:41] ok milimetric :] [15:51:43] schema_version? [15:52:36] Analytics-Cluster, Analytics-Kanban: Refactor MobileApps uniques HQL to use external table to format data. - https://phabricator.wikimedia.org/T90730#1065989 (JAllemandou) NEW [15:52:47] Yup, it's a good idea [15:52:55] I'll do that on different branch [15:53:22] I prefer record_version [15:53:45] ok cool [15:54:03] das good for me [15:56:35] mforns: I will let milimetric review your changes for the generator code. [15:56:59] nuria, sure, I just wanted to send the corrected link [16:07:38] !log hello? [16:17:33] Analytics-Cluster, Analytics-Kanban: Refactor MobileApps uniques HQL to use external table to format data. - https://phabricator.wikimedia.org/T90730#1066047 (Ottomata) If we do this, I think we should add the date information to each file. I'm fixing the bug where the files were named incorrectly, and I... [16:32:48] mforns, milimetric : re-started wikimetrics [16:33:01] nuria, what happened? [16:33:24] mforns: yesterday there was a lab outage rememeber? [16:33:28] yes [16:33:40] mforns: and normally after one of those the tool needs a re-start [16:33:56] I see, ok [16:35:16] mforns, milimetric , kevinator : to permanently fix pageviews in dashiki a puppet change is needed [16:36:01] thx nuria, kevinator should prioritize but he's not on right now, I'll send an email [16:36:55] nuria: btw. I followed up with Amanda yesterday, she was ok [16:37:56] milimetric: ah , ok, let's please try to cclist when we communicate with users so as not to repeat ourselves [16:38:33] well, she dropped a lot of people from the cc, and I think it's bad email etiquette to re-add [16:39:10] I kind of disagree with our unmanageable emailing practices, cc-ing everyone for Everything [16:39:23] hey joal, yt? [16:41:26] milimetric: but not everyone, just Wikimetrics@ that is what we have the list for, right? [16:42:20] hey ottomata [16:42:37] hey, hm, nm, i was going to ask you to test something with spark, but i just realized I only fixed it for pyspark [16:42:41] trying with spark-shell now [16:42:44] Oh! [16:42:46] maybe. hang on [16:43:21] yes ok [16:43:31] tell me [16:43:31] joal, can you try your spark-shell thing with the raw webrequest data [16:43:35] before starting shell [16:43:36] do [16:43:38] export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native [16:44:40] First good sign : no warning at startup :) [16:45:18] i've checked that i don't get an exception with pyspark whne using sparkcontext sequenceFile [16:45:26] that's as far as I've gone :) [16:51:38] joal, ees ok? [16:53:47] S'perfect man !!! [16:53:56] Happy joal 1 [16:54:30] I guess you'll automate the setting with puppet .? [16:58:21] ja will do [16:58:28] You goooooood :D [17:16:16] Analytics-Wikistats, I18n, LE-Sprint-82, Patch-For-Review: stats.wikimedia.org - the "User" namespace for Nynorsk is wrong - https://phabricator.wikimedia.org/T89387#1066158 (Amire80) [17:27:32] (PS1) Joal: Add record_version field to refined webrequest table. [analytics/refinery] - https://gerrit.wikimedia.org/r/192824 [17:29:09] ottomata: do you have any idea as to the varnish version we are running in prod? [17:30:05] Analytics-Cluster, Analytics-Kanban: Add jar versions as parameters in oozie jobs - https://phabricator.wikimedia.org/T90736#1066190 (JAllemandou) NEW [17:34:01] (CR) Nuria: Add record_version field to refined webrequest table. (1 comment) [analytics/refinery] - https://gerrit.wikimedia.org/r/192824 (owner: Joal) [17:44:19] Analytics-Dashiki, Analytics-Kanban: Pageviews not loading in Vital Signs - https://phabricator.wikimedia.org/T90742#1066310 (kevinator) NEW [17:46:01] Analytics-Dashiki, Analytics-Kanban: Pageviews metric not showing in Vital Signs - https://phabricator.wikimedia.org/T90587#1066340 (kevinator) [17:46:01] Analytics-Dashiki, Analytics-Kanban: Pageviews not loading in Vital Signs - https://phabricator.wikimedia.org/T90742#1066341 (kevinator) [17:51:54] Analytics: What is the bounce rate of wikipedia.org as compaired to xx.wikipedia.org - https://phabricator.wikimedia.org/T90743#1066379 (Jaredzimmerman-WMF) NEW [17:58:17] (CR) Ottomata: [C: 1] Add record_version field to refined webrequest table. (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/192824 (owner: Joal) [18:13:04] Analytics-EventLogging, Analytics-Kanban, Mobile-Web: Follow up with mobile team on instrumentation sampling rate (%50) - https://phabricator.wikimedia.org/T88363#1066439 (KLans_WMF) a:Milimetric>kaldari [18:18:55] nuria: did the backfill of events on vanadium finish? [18:19:41] ori: other than the raw events mostly yes [18:19:59] so -- can we update vanadium to master? [18:20:49] ori: no, not yet, this would need to be merged: https://gerrit.wikimedia.org/r/#/c/191231/ [18:21:45] ori: I am going to further test that patch with the backfilling of raw events (there were two different outages, one due to capsule changes, other due to db not keeping up) [18:21:52] ori: im getting you an eventlogging node today =] [18:23:14] ori: since teh backfilling backfills events 1 by 1 and it takes 10 sec to backfill ~1000 it's not the speediest process [18:23:21] nuria, ottomata link to both places ? [18:23:36] ori: nice, when do we do the switch then [18:23:49] joal: whatever ottomata suggests is good [18:23:57] k thx [18:25:55] joal: don't care about both places, but def in the table comment [18:26:02] so people can see it when they run [18:26:05] describe webrequest; [18:26:10] yup [18:26:12] sounds good [18:26:38] (CR) Milimetric: "I only got partially through, and left some comments. I have to leave it at this as I have to focus on other stuff." (10 comments) [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [18:29:18] Quick question : when updating a review, should I commit on top of the initial one, or squash ? [18:29:33] joal: you can do commit --amend [18:29:45] then git review ? [18:29:48] so you are comitting over & over the same changeset [18:30:05] and gerrit knows about the multiple patches ? [18:30:08] joal: It will be easier to do git push origin HEAD:refs/for/master [18:30:32] joal: no need to involve git review [18:30:49] joal: gerrit looks at change-id [18:30:54] joal: and that remains [18:31:00] joal: makes sense? [18:31:05] pssh git review is cool [18:31:08] don't listen to nuria :p [18:31:09] yes [18:31:12] git commit --amend [18:31:14] and then git review [18:31:20] huhu :) [18:31:22] ottomata: jaja, this comes from qchris, ah [18:31:29] taka "the master of gerrit" [18:31:31] as long as the Change-Id on the commit message of HEAD is the same [18:31:34] "aka" [18:31:39] git review will submit it as a new patchset on the same change in gerrit [18:31:57] I'll take the easy one for now, but will definitely try to learn more about gerrit internals in the future :) [18:32:02] Thx folks [18:32:07] joal: k [18:34:10] (PS2) Joal: Add record_version field to refined webrequest table. [analytics/refinery] - https://gerrit.wikimedia.org/r/192824 [18:35:07] ori: when should we plan to switch the box? like next week? tomorrow? (kidding) [18:36:03] (CR) Ottomata: [C: 2 V: 2] Add record_version field to refined webrequest table. [analytics/refinery] - https://gerrit.wikimedia.org/r/192824 (owner: Joal) [18:38:09] milimetric: SoS? [18:38:27] no, I have brain bankrupcy [18:38:29] I can't do it [18:38:31] too much! [18:38:34] did you find a sub?! [18:39:22] Analytics, Mobile-Apps, Wikipedia-App-Android-App, Wikipedia-App-iOS-App, and 2 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1066600 (dr0ptp4kt) [18:40:00] ugggggggg [18:40:41] haha [18:40:51] let me know if you need some counseling [18:42:10] i totally do, much counseling... [18:46:30] Analytics, Mobile-Apps, Scrum-of-Scrums, Wikipedia-App-Android-App, and 3 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1066658 (dr0ptp4kt) [18:48:17] (CR) Mforns: "Thanks for the review! Answered to your comments. In short will push the changes needed." (9 comments) [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [18:57:51] yurik: i am here [18:57:51] https://plus.google.com/hangouts/_/wikimedia.org/a-batcave [19:06:10] Analytics-Cluster, Analytics-Kanban: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1066753 (kevinator) NEW [19:06:44] yurik: I just created the task ^^ and added link to etherpad [19:06:59] thx kevinator !!! [19:08:40] Analytics-Cluster, Analytics-Kanban: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1066770 (Milimetric) privacy implications must be very carefully considered here. The current reports that are created by Erik Z. take a lot of care to add fuzziness wher... [19:24:46] Analytics-Cluster, Analytics-Kanban: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1066914 (Nuria) I think we need an example of how does the report look like. Are pageviews per page? Per project? Per language? [19:24:52] Analytics-Engineering, Analytics-EventLogging, Analytics-Kanban: Spike on requirements to prune EL data {oryx} - https://phabricator.wikimedia.org/T89293#1066915 (kevinator) This could be discussed in an email thread [19:30:11] Analytics-EventLogging, Analytics-Kanban: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1066942 (kevinator) [19:30:27] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1066944 (Krenair) [19:39:13] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1066980 (kevinator) [19:40:56] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1066936 (kevinator) I'm probably going to revisit this for Q4 planning. not right now :-) [19:41:27] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1067000 (kevinator) a:kevinator [19:42:28] Analytics-EventLogging, Analytics-Kanban, Documentation, Epic: {epic} Product Instrumentation and Visualization {oryx} - https://phabricator.wikimedia.org/T76795#1067006 (kevinator) [19:42:29] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1066936 (kevinator) [19:48:41] Analytics-Engineering, Analytics-EventLogging: Investigate EventLogging Monitoring with Ops DBA - https://phabricator.wikimedia.org/T86200#1067037 (kevinator) Here is a message from qchris way back on Jan 6 2015 > A pending thing I can think of would be to transfer the monitoring you have > regarding rep... [19:49:26] Analytics-EventLogging, Analytics-Kanban: Investigate EventLogging Monitoring with Ops DBA - https://phabricator.wikimedia.org/T86200#1067038 (ggellerman) [19:52:08] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1067050 (Tgr) > Schema errors are hard to notice - they are logged to the console but tend to get los... [19:56:53] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1067071 (Nuria) @tgr: But you know that schema validation is real easy to test in vagrant with the d... [19:57:42] Analytics-EventLogging, Analytics-Kanban, MediaWiki-extensions-MultimediaViewer, Multimedia: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#1067081 (Nuria) @tgr: But you know that schema validation is real easy to test in vagrant with the d... [20:18:33] oh man oh man [20:18:34] so awesome [20:18:35] http://confluent.io/docs/current/platform.html [20:18:48] if we were starting to build this cluster now, i'd probably just use this [20:22:33] is it all puppetized and debianized and everything? Looks like good architecture [20:23:28] not puppetied [20:23:31] but debianized, yes [20:23:50] next time i have to upgrade kafka, i will investigate what is needed to use their kafka deb package [20:23:54] would much rather not maintain that [20:24:03] but, they have camus packaged [20:24:08] and the schema registry! [20:24:13] oh man, that wasn't even really avaialbe before, afaik [20:24:18] Analytics-Cluster, Analytics-Kanban: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1067308 (Yurik) [20:24:56] and a built in kafka rest interface [20:24:58] yep, I agree with Toby that the schema reg. is very interesting [20:25:06] right! I liked that too [20:25:14] especially on the consumer side [20:25:14] i mean, we kiinda have that with kafka :p [20:25:19] well, sorry [20:25:20] i'm not sure how that works but that sounds great! [20:25:24] we can produce via http via varnishkafka [20:25:25] but ja [20:25:54] greenfield during the hackathon if you want to play with it :) [20:27:12] haha [20:27:21] naw, i mean, naw, i do, but I already have an idea [20:27:38] hm, i wonder if the rest interface would allow you to transparently read avro data from kafka using the schema registry [20:27:40] that would be SO AWESOME [20:27:54] that would mean that people could consume avro from kafka and get it in json [20:33:37] Analytics-Cluster, Analytics-Kanban: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1067360 (Yurik) Per T90499, for Zero, we would need a table with the following columns: language, subdomain (nothing|m|zero), site (wikipedia|...), country, count, bandw... [20:41:48] Analytics-Cluster, Analytics-Kanban, Scrum-of-Scrums: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1066753 (Yurik) [20:54:41] Analytics, MediaWiki-extensions-MultimediaViewer, Multimedia, Multimedia-Sprint-2015-02-25, Patch-For-Review: Set up varnish 204 beacon endpoint for virtual media views and use it in Media Viewer - https://phabricator.wikimedia.org/T89088#1067420 (Gilles) [21:17:03] Analytics-Engineering: EPIC: Getting Mondrian & Saiku productionized - https://phabricator.wikimedia.org/T76739#1067560 (kevinator) This project is to be named {puma} [21:17:33] Analytics-Kanban: EPIC: Getting Mondrian & Saiku productionized {puma} - https://phabricator.wikimedia.org/T76739#1067568 (kevinator) p:Normal>Low [21:17:55] Analytics-Kanban: Getting Mondrian & Saiku productionized {epic} {puma} - https://phabricator.wikimedia.org/T76739#819159 (kevinator) [21:27:02] (PS1) Joal: Add refinery_hive_jar_version as a parameter in oozie bundle properties. [analytics/refinery] - https://gerrit.wikimedia.org/r/192891 [21:29:12] Analytics, Fundraising Sprint Enya, Fundraising Tech Backlog, Wikimedia-Fundraising: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1067644 (awight) @ellery: any results? [21:35:07] Tomorrow guys :) [21:39:27] (PS6) Mforns: [WIP] [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) [21:39:34] (CR) jenkins-bot: [V: -1] [WIP] [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [21:39:44] (PS9) Jsahleen: Reports: Add reporting system for generating limn sql [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/192457 (https://phabricator.wikimedia.org/T90265) [21:40:06] (CR) Mforns: "Dan, still working on your comments. I'll ping you when done." [analytics/limn-mobile-data] - https://gerrit.wikimedia.org/r/192319 (https://phabricator.wikimedia.org/T89251) (owner: Mforns) [21:41:09] (PS10) Jsahleen: Reports: Add reporting system for generating limn sql [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/192457 (https://phabricator.wikimedia.org/T90265) [21:42:01] mforns: no worries, the only reason I submitted my comments half-done was to unblock you [21:42:27] milimetric, ok [21:42:40] (CR) Jsahleen: [C: 2] "Implemented suggestions from Niklas." [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/192457 (https://phabricator.wikimedia.org/T90265) (owner: Jsahleen) [21:42:44] I have still things to do, so... not blocked at all :] [21:55:26] kevinator: https://edit-analysis.wmflabs.org/adhoc.html#ve-overall-success-by-user-types.tsv [21:58:01] (PS1) Milimetric: [WIP] Analyze edit success rate by user type [analytics/limn-edit-data] - https://gerrit.wikimedia.org/r/192944 (https://phabricator.wikimedia.org/T89729) [22:28:25] Analytics-Cluster, Analytics-Kanban, Easy: Mobile Apps PM has monthly report from oozie about apps uniques - https://phabricator.wikimedia.org/T88308#1067855 (kevinator) p:Normal>High [22:35:18] Analytics, Engineering-Community, ECT-February-2015: Analytics Team Offsite - Before Wikimania - https://phabricator.wikimedia.org/T90602#1067898 (Rfarrand) Met today with Toby and Leila, will get back to them with a few options soon! [22:48:01] ottomata: You asked "hello?" today on 16:07 (which I logged at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log ) ... so ... hello :-) [22:48:30] ahah [22:48:39] :-D [22:49:46] I am not sure about this bot. The log target is documented on the bot's page. [22:49:56] But I also never found it. [23:03:21] I've got a hive job which has been stalled 4 ever... if anyone can take a look or inform of general downtime? Starting Job = job_1424120984454_19292, Tracking URL = http://analytics1001.eqiad.wmnet:8088/proxy/application_1424120984454_19292/ [23:03:25] Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1424120984454_19292 [23:04:09] wikimedia/mediawiki-extensions-EventLogging#358 (wmf/1.25wmf19 - e3d6557 : Mukunda Modell): The build passed. [23:04:09] Change view : https://github.com/wikimedia/mediawiki-extensions-EventLogging/commit/e3d6557a7f43 [23:04:09] Build details : http://travis-ci.org/wikimedia/mediawiki-extensions-EventLogging/builds/52177901 [23:06:20] awight: query? [23:06:36] your job hasn't even started [23:06:48] that's what I thought... [23:06:57] is there a way mortals can see that sort of thing? [23:09:03] what was the query you ran? [23:09:33] Analytics, Fundraising Sprint Enya, Fundraising Tech Backlog, Wikimedia-Fundraising: Strategy banner impressions - https://phabricator.wikimedia.org/T90635#1068062 (awight) Trying this one (still waiting for it to start): ``` select count(*), parse_url(concat('http://bla.org/woo/', uri_qu... [23:09:44] ottomata: https://phabricator.wikimedia.org/T90635#1068062 [23:09:44] awight, also [23:09:46] https://hue.wikimedia.org/jobbrowser/ [23:09:52] log in with your shell username and ldap pw [23:09:59] badass [23:19:47] hmmm, something is def wrong awight... [23:21:32] jobs are backed up all over the place [23:21:33] uh oh [23:22:50] this is really weird [23:22:51] ottomata: I'd like to ask ewulczyn about this too, but in general, I don't want to be making Hive queries for this stuff. Is there a precedent for jobs that periodically distill Hive data into a more real-time data store? [23:23:18] awight, let's talk about this, yes/no, you are doing fundraising stuff, right? [23:23:25] yes [23:23:30] let's talk later though, cuz uh, something is really broken [23:23:40] no jobs completed since this mornin [23:23:43] morning [23:24:14] yah this is a long-term thing, don't worry now! Here's the task if you're curious. https://phabricator.wikimedia.org/T90649 [23:25:32] haha [23:25:47] awight, quick answer, your options are either kafkatee [23:25:58] https://github.com/wikimedia/analytics-kafkatee [23:26:03] which gives you a udp2log like interface on this data [23:26:05] or [23:26:09] something fancier that i don't know how to use yet [23:26:16] we should ahve spark streaming available if you want to try you rluck [23:26:35] well, I *think* we do, not sure yet [23:27:59] YARGHGHHH [23:28:03] bad timign i really have to goooOOoo [23:28:27] ottomata: Same thing happened yesterday, which made me stay up until 5am :-( [23:28:33] The cluster is overloaded. [23:28:49] ? [23:28:51] Some fancy query grabs all resources it can. [23:28:55] you stayed up til 5am? [23:28:58] But wants to grab more. [23:29:01] i noticed a job that had been restarted [23:29:03] we should talk about this! [23:29:04] :) [23:29:19] qchris, i did make a change yesterday that increased the number of available vcores [23:29:41] Meh. I stopped billing wmf some time ago. It's ... community support for me. [23:29:48] haha [23:29:55] crap crackers [23:30:02] i mean, it is possible my c hange caused this [23:30:17] but i would have thought this would have given us more capacity, maybe it just made yarn schedule mroe stuff at once than it can handle [23:30:35] Maybe. [23:32:08] Analytics, Wikimedia-Fundraising: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1068169 (awight) @ottomata mentioned that we could look at https://github.com/wikimedia/analytics-kafkatee My preference is definitely to run some logic as part of the... [23:36:24] !log freed up cluster resources to allow camus to deliver its content [23:52:12] yurik you still around? [23:52:21] Mhmm.... no yurik.