[07:22:30] (Draft2) MarcoAurelio: Configuration for olo.wikipedia.org [analytics/refinery] - https://gerrit.wikimedia.org/r/312810 [07:22:37] (Draft1) MarcoAurelio: Configuration for olo.wikipedia.org [analytics/refinery] - https://gerrit.wikimedia.org/r/312810 [07:28:05] (PS3) MarcoAurelio: Configuration for olo.wikipedia.org [analytics/refinery] - https://gerrit.wikimedia.org/r/312810 (https://phabricator.wikimedia.org/T146612) [07:37:48] (CR) MarcoAurelio: "check experimental" [analytics/refinery] - https://gerrit.wikimedia.org/r/312810 (https://phabricator.wikimedia.org/T146612) (owner: MarcoAurelio) [07:37:57] Hey, ottomata at european time !! [07:38:03] hello :) [07:38:10] hellOoooo! [07:38:58] joal_: sounds like we won't be doing the remote overview of mw history via hangout on thurs [07:39:08] okey [07:39:29] too many things to do I guess :) [07:39:40] well, its just the hangout part that is hard it hink [07:39:42] so oOooO [07:39:51] i'm going to find some time to talk yall about what luca and I should stay! [07:40:01] say* [07:40:18] sounds good :) [07:42:40] o/ [07:42:59] Hi elukey :) [09:06:50] hey joal_ ! [09:34:05] Hi addshore [09:44:29] (CR) Addshore: WikidataArticlePlaceholderMetrics also send search referral data (1 comment) [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) (owner: Addshore) [09:44:57] joal_: :) I was going to ask a question / opinion about ^^ but I think I have it figured out :) [09:45:05] but a review would be great! :D [09:55:24] addshore: reading / testing [09:55:38] thanks! [10:09:21] addshore: how have you performace tested you requests? [10:13:01] So I tested the one currently in that patch through the whole process spark etc. [10:13:35] and earlier today I checke da few variations looking at the referrer and ignoring emptys but just thought hive [10:13:39] *through [10:13:47] k [10:14:34] addshore: testing for performance on the cluster can only be done when looking at big perf differences [10:14:54] there are many reasons for which the same query would actually take more or less time [10:15:20] As an exemple, I ran the patch's query 2 times in a row, and got 2 different perf results [10:15:31] Difference is not huge, but it is still present [10:15:35] addshore: --^ [10:16:15] Now, for regexp, tests should be done in scala (or java), locally - where things are compatrable :) [10:16:15] yeh, I ran each version 3 times I think, getting slightly different cpu times for each, and even slightly different hdfs reads [10:16:32] addshore: right [10:16:55] addshore: I have a suggestion for a big improvement though :) [10:17:15] addshore: You could easily run the 2 queries you have as a single one, reconciliating values in scalab [10:17:33] Like this, you split the computation by 2 [10:18:48] hmm, makes sense! ;) [10:19:04] I would probably get a bit lots in that hive query! [10:19:49] addshore: Since the cardinality of the dimensions you are retrieving is small enough, the global number of values if running everything in a single query is not too bigeither [10:20:35] addshore: I'm sure you won't :) [10:20:38] addshore: [10:20:45] addshore: https://gist.github.com/jobar/c2cb53cf70a0fb1b9892520a6160de77 [10:20:48] sorry [10:21:46] addshore: you collect the result, and then sum-again by project and needed dimensions [10:24:37] addshore: another way in a new comment on the gist [10:24:46] addshore: very similar though :) [10:25:19] oooh, okay! [10:25:36] addshore: You pick the one you like ;) [10:26:13] (CR) Addshore: [C: -1] "I'm going to revist this and do something like:" [analytics/refinery/source] - https://gerrit.wikimedia.org/r/305989 (https://phabricator.wikimedia.org/T142955) (owner: Addshore) [11:15:17] Analytics-Tech-community-metrics: Two "Submitters" widgets in Kibana show no data: "Could not locate that index-pattern-field (id:project)" - https://phabricator.wikimedia.org/T146629#2666749 (Aklapper) [11:15:31] Analytics-Tech-community-metrics: Two "Submitters" widgets in Kibana show no data: "Could not locate that index-pattern-field (id:project)" - https://phabricator.wikimedia.org/T146629#2666749 (Aklapper) p:Triage>High [11:17:29] Analytics-EventLogging, Continuous-Integration-Config: EventLogging CI jobs hit meta.wikimedia.org - https://phabricator.wikimedia.org/T122463#2666774 (hashar) Open>declined Apparently does not cause much harm. [11:20:58] Analytics-Tech-community-metrics: Empty "Authors by the Earliest Commit" widget in Kibana's "Git-Demographics" - https://phabricator.wikimedia.org/T146630#2666788 (Aklapper) [11:21:01] Analytics-Tech-community-metrics: Data in "The Newest Authors" widget in Kibana's "Git-Demographics" is not updated - https://phabricator.wikimedia.org/T146631#2666799 (Aklapper) [11:21:11] Analytics-Tech-community-metrics: Data in "The Newest Authors" widget in Kibana's "Git-Demographics" is not updated - https://phabricator.wikimedia.org/T146631#2666799 (Aklapper) p:Triage>High [11:21:49] Analytics-Tech-community-metrics: Empty "Authors by the Earliest Commit" widget in Kibana's "Git-Demographics" - https://phabricator.wikimedia.org/T146630#2666788 (Aklapper) p:Triage>Lowest (Setting priority to lowest as I'm not convinced that this widget is useful for us.) [11:26:38] Analytics-Tech-community-metrics: Kibana's Mailing List data sources are outdated - https://phabricator.wikimedia.org/T146632#2666814 (Aklapper) [11:26:46] Analytics-Tech-community-metrics: Kibana's Mailing List data sources are outdated - https://phabricator.wikimedia.org/T146632#2666814 (Aklapper) p:Triage>High [11:27:06] Analytics-Tech-community-metrics, Developer-Relations (Jul-Sep-2016): Identify Wikimedia's most important/used info panels in korma.wmflabs.org - https://phabricator.wikimedia.org/T132421#2666826 (Aklapper) [12:14:56] hi team! [12:15:03] Hey mforns [12:15:13] hello joal_! [12:15:14] mforns: When you have a minute I have questions for you :) [12:15:19] now? [12:15:23] batcave! [12:15:27] sure ! [13:42:38] morning yall [13:45:55] heya [13:46:00] taking a break before standup [14:43:31] PROBLEM - Difference between raw and validated EventLogging overall message rates on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [30.0] [14:53:31] RECOVERY - Difference between raw and validated EventLogging overall message rates on graphite1001 is OK: OK: Less than 20.00% above the threshold [20.0] [15:02:28] Analytics-Kanban: Upload the final version of the pivot repo and load test - https://phabricator.wikimedia.org/T146389#2667429 (mforns) a:Milimetric>mforns [15:03:07] Analytics-Kanban: Spamy - User-like pages that should not be allowed to be created are hit by bots and distort our pageview metrics (they return 200) - https://phabricator.wikimedia.org/T145922#2667431 (Nuria) [15:03:38] Analytics-Kanban: Spamy - User-like pages distort our pageview metrics (they return 200 when they should return 404) - https://phabricator.wikimedia.org/T145922#2645266 (Nuria) [15:04:01] Analytics-Kanban: Spamy - User-like pages distort our pageview metrics (they return 200 when they should return 404) - https://phabricator.wikimedia.org/T145922#2645266 (Nuria) a:Nuria [15:25:35] mforns: http://www.gingersoftware.com/content/grammar-rules/adjectives/order-of-adjectives/ [15:28:39] o/ milimetric, joal_ & schana [15:28:48] Hi halfak [15:28:51] Looking at the live systems meeting scheduling [15:28:55] I figured IRC would be easier. [15:29:10] Since joal_ can't make the next two on Weds, I was going to suggest we just do Thurs instead. [15:29:14] Long term [15:31:24] joal: grooomingg? [15:31:34] halfak: I can make next thursday, not in 2 weeks, but globally yes [15:31:48] nuria_: arf, scuse me [15:31:50] joining [15:31:51] halfak: anything works for me, my calendar's up to date [15:32:01] Sorry nuria_. my fault :/ [15:32:03] Thanks milimetric [15:32:09] schana? [15:33:15] halfak: don't worry [15:34:54] Analytics: Load storage (druid? clickhouse?) with calculated edit metrics data and serve under an external endpoint - https://phabricator.wikimedia.org/T146490#2667511 (Nuria) [15:38:08] Analytics: Improve mediawiki data redaction and refactor edit history reconstruction - https://phabricator.wikimedia.org/T146444#2667527 (Nuria) [15:45:09] Analytics, Analytics-Visualization, Reading-Admin: Mobile PMs has visualization on session-related metrics from Wikipedia Apps - https://phabricator.wikimedia.org/T94481#2667538 (Nuria) Open>Resolved a:Nuria Old ticket, data is available and if you want to visualize them please reopen. [15:46:51] Analytics, Analytics-Wikistats: Design new UI for Wikistats 2.0 - https://phabricator.wikimedia.org/T140000#2667543 (Milimetric) [15:46:53] Analytics: Dashboard Directory Design Feedback - https://phabricator.wikimedia.org/T92502#2667542 (Milimetric) [15:46:59] Analytics: Dashboard Directory Design Feedback - https://phabricator.wikimedia.org/T92502#1113178 (Milimetric) p:Triage>Normal [15:50:47] Analytics, Analytics-EventLogging, MediaWiki-extensions-MultimediaViewer, Reading-Web-Backlog: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#2667554 (Nuria) Many of these features (like sampling) already exist on EL client. Also, testing... [15:51:04] Analytics, Analytics-EventLogging, Documentation, Epic: Product Instrumentation and Visualization {oryx} - https://phabricator.wikimedia.org/T76795#2667557 (Nuria) [15:51:07] Analytics, Analytics-EventLogging, MediaWiki-extensions-MultimediaViewer, Reading-Web-Backlog: Parse mediaviewer team's requirements for EventLogging {oryx} - https://phabricator.wikimedia.org/T90766#2667555 (Nuria) Open>Resolved a:Nuria [15:53:16] Analytics, Analytics-Cluster: generate monthly Pageview cubes - https://phabricator.wikimedia.org/T95505#2667564 (Nuria) Open>Resolved a:Nuria Done in Druid. This data is available in Druid, accessible via pivot UI (available on Q2) . WOOWWW!!! [15:55:54] Analytics, Analytics-Cluster: Estimate roughly of how many users might not have javascript capable/enable browsers, use CSS to crosscheck. - https://phabricator.wikimedia.org/T89847#2667572 (Nuria) Open>Resolved a:Nuria We run rough estimates a while back (2 years). Nobody needs an update th... [15:58:56] Analytics: Publish aggregate geodumps of article pageviews - https://phabricator.wikimedia.org/T91331#2667587 (Nuria) Open>Resolved [16:00:06] Analytics: Analyze difference in Edit Schema "bounce rates" across wikis {lion} - https://phabricator.wikimedia.org/T89726#1043805 (Milimetric) @HJiang-WMF: we found this in our old tasks. Let me know if you're interested in looking at it and I can explain more. Basically, James was wondering how bounce ra... [16:00:23] Analytics, Fundraising-Analysis: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2667594 (Nuria) Open>Resolved a:Nuria I think issues on ticket have solved, let us know otherwise. [16:02:28] Analytics: Unique devices endpoint Graphana Dashboard {bear} - https://phabricator.wikimedia.org/T132795#2667602 (Nuria) Open>Resolved a:Nuria Can be closed, already done. [16:03:43] Analytics, Analytics-Dashiki: Add time range selection to Dashiki dashboards - https://phabricator.wikimedia.org/T87603#2667619 (Nuria) [16:04:54] Analytics, Analytics-Dashiki: Add time range selection to Dashiki dashboards - https://phabricator.wikimedia.org/T87603#994953 (Nuria) Open>Resolved Dashiki Dashboards let you zoom in time. We are no longer updating limn. [16:27:51] (CR) Nuria: [C: 2 V: 2] "Please note this change does not get immediately deployed. Thank you." [analytics/refinery] - https://gerrit.wikimedia.org/r/312810 (https://phabricator.wikimedia.org/T146612) (owner: MarcoAurelio) [16:33:28] Quarry: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#2667746 (Milimetric) @Quiddity, we're very close to allowing this kind of query in Hadoop, which you've worked with before, right? Grab me and I'll show you what we're working... [16:44:37] elukey: looking at EL alarms [16:57:15] Analytics, Research-and-Data, Research-collaborations, Research-management, Patch-For-Review: Oozie job to extract data for WDQS research - https://phabricator.wikimedia.org/T146064#2667958 (leila) >>! In T146064#2662464, @Nuria wrote: > @leila: none of our datasets captures a raw IP due to p... [17:08:43] Analytics, Fundraising-Analysis: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2667994 (awight) Resolved>Open Unfortunately, I don't know of any progress towards this goal. We still need to run the expensive Hive query, repeatedly parsing... [17:09:15] Analytics, Fundraising-Analysis, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#2667998 (awight) [17:09:52] Analytics, Fundraising-Analysis, Fundraising-Backlog, MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1064105 (awight) (see also T115042) [17:19:44] Analytics-Kanban: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668022 (Nuria) [18:03:56] Analytics-Kanban: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668189 (Nuria) Since today's at 6:30 am this is the split of client parsing errors: 1 "MultimediaViewerDuration" 1 "NavigationTiming" 1 "PageD cp1067.eqiad.wmnet 42439819 2016-09-26T12 2 "Centr... [18:13:50] going off for tonight a-team [18:13:56] good night joal_! [18:14:04] mforns: keeping you posted, I have a first draft seemingly working [18:14:06] nite [18:14:21] mforns: Currently testing it agaginst enwiki [18:14:31] joal_, awesome! [18:14:39] mforns: simplewiki run-time got really improved [18:14:46] cool! [18:14:52] o/ [18:14:58] So I have good feelings ;) [18:15:03] Bye ! [18:30:35] mforns: we should review the ones you're not sure about tomorrow the same way we did with mine today [18:30:47] milimetric, aha [18:30:49] so keep tagging me and nuria, that works [18:30:53] cool [18:38:18] milimetric, mforns : will look at those later on today, after i digg on EL a bit more [18:39:35] cool nuria_ [18:40:27] k [18:40:37] nuria_: let us know if we can help with EL troubleshooting [18:40:48] milimetric: k [18:41:00] I'm probably going to screen resumes full time since that dashiki task was my last real thing for the quarter [18:41:17] when I'm finished with resumes I'll go document and clean things up, but I figure resumes are a priority now [18:46:50] milimetric: you know how to look at your localhost in your ipad? [18:46:58] milimetric: you can also test the layout that way. [18:47:24] milimetric: if you do http://your-ip-in-network:port you can access dashiki and see the real look on device [18:48:13] nuria_: yeah, but isn't the device mode a decent simulation? [18:48:32] milimetric: no, cause there are no touch events [18:48:43] nuria_: with that, I only spent a couple minutes looking at it, the main thing is the new layout framework [18:48:57] with the new framework we can do anything we want, so that's what I was excited about [18:49:03] milimetric: right, right, that looks wayyy wayy beter [18:49:26] milimetric: like ahem.. a million times [18:49:53] nuria_: I see, yeah, the touch stuff needs fixing [18:50:17] milimetric: cause you cannot click on menus , well, it is on and off [18:50:43] yeah, that's semantic [18:51:03] milimetric: i will document it, but I did not want to imply you need to do anything else [18:51:03] I changed it a bit to make it simpler, because the old way was so hacky it broke completely in the new version [18:51:18] no, I knew that'd be more broken now in mobile [18:51:24] milimetric: just that testing on device is actually pretty easy [18:51:25] it's sad, but necessary, we'll fix it eventually [18:51:33] yep [18:51:41] milimetric: ya, i will document and we can fill in bugs as needed. [18:52:19] Analytics-Kanban: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668349 (Nuria) On prior couple days: 2 "EchoInteraction" 2 "PrefUpdate" 3 "MobileWikiAppShareAFact" 4 "Search" 5 "NavigationTiming" 5 "WikipediaPortal" 6 "MobileAppUploadAtt... [18:58:20] (PS1) MaxSem: Fix syntax error [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/312858 (https://phabricator.wikimedia.org/T146592) [19:03:13] (CR) Yurik: [C: 2] Fix syntax error [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/312858 (https://phabricator.wikimedia.org/T146592) (owner: MaxSem) [19:11:44] Analytics-Kanban: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668409 (Nuria) Day before: 2 "EchoInteraction" 2 "NavigationTiming" 3 "Search" 4 "MobileAppUploadAttempts" 4 "ServerSideAccountCreation" 6 "MultimediaViewerNetworkPerformance"... [19:12:10] Analytics: passport-mediawiki-oauth doesn't support callback parameter - https://phabricator.wikimedia.org/T145828#2668410 (Jdlrobson) Digging around this seems to be due to the workaround which I suspect is causing this problem: ``` this._oauth._authorize_callback = 'oob'; ``` passport-oauth sets: ``` para... [19:21:37] Analytics-Kanban: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668434 (Nuria) Pinging moriel on this ticket. @MSchottlender-WMF There is a pretty big number of events for event logging schema: https://meta.wikimedia.org/wiki/Schema:ChangesListFilters That are not validatin... [19:22:00] Analytics-Kanban, Community-Tech: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668436 (Nuria) [19:22:35] mforns: eventlogging stuff mostly done, looking to greenhose now [19:22:40] *greenhouse [19:33:54] (CR) MaxSem: [V: 2] Fix syntax error [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/312858 (https://phabricator.wikimedia.org/T146592) (owner: MaxSem) [19:34:56] (PS1) MaxSem: Fix syntax error [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/312863 (https://phabricator.wikimedia.org/T146592) [19:35:02] (CR) MaxSem: [C: 2 V: 2] Fix syntax error [analytics/discovery-stats] - https://gerrit.wikimedia.org/r/312863 (https://phabricator.wikimedia.org/T146592) (owner: MaxSem) [19:47:16] Analytics-Kanban, Community-Tech: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668507 (kaldari) @Mooeypoo: Looks like whatever is trying to use the ChangesListFilters EventLogging schema is failing to pass the page namespace consistently. [20:24:57] Analytics: passport-mediawiki-oauth doesn't support callback parameter - https://phabricator.wikimedia.org/T145828#2668625 (Tgr) You are supposed to pass `oob` if and only if the consumer does not have the "Allow consumer to specify a callback..." checkbox checked (see [[https://oauth.net/core/1.0a/#auth_ste... [21:07:37] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668862 (kaldari) [21:14:12] omg it's so hard looking at nothing but applications all day long [21:14:16] I'm so tired :) [21:16:49] Analytics-Kanban, MediaWiki-extensions-WikimediaEvents, Collab-Team-Q1-July-Sep-2016: EL alarms raw/validated 20160926 - https://phabricator.wikimedia.org/T146674#2668873 (Mooeypoo) a:Mooeypoo Okay, I was under the impression that since I tagged the namespace value as 'optional', the logger will... [21:21:49] Quarry: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#2668912 (yuvipanda) Hadoop isn't public though, so not the same thing :) You can do similar things via tool labs now too. [21:33:01] Quarry: it would be useful to run the same Quarry query conveniently in several database - https://phabricator.wikimedia.org/T95582#1195035 (matmarex) You can fairly easily turn a query for a single wiki into a query over all wikis. It's a bit of boring work though. The pattern goes like this: ```lang=sql s...