[07:37:25] Aww, AdBlockPlus now blocks charts at https://analytics.wikimedia.org/dashboards/vital-signs/ [07:43:03] \o/ [07:43:16] I blame the analytics folks [08:42:18] Hi elukey [08:42:22] o/ [08:42:43] emails and some orga things, then cluster deploy + uniques productionisation if you're ok [08:59:10] sure! [08:59:17] I am going to commute to the office soon [08:59:27] but in max 30 mins I'll be online [08:59:32] (if you need me) [09:00:06] noted elukey, I'll for you to be back [09:00:26] * joal prefers when elukey is here before touching production stuff [09:09:43] 10Analytics-Kanban: Document that old deleted pages have empty fields in Analytics Cluster edit data - https://phabricator.wikimedia.org/T165201#3339493 (10JAllemandou) >>! In T165201#3336156, @mforns wrote: > Wrote this, I hope it's enough to have it in the Data_Lake/Edits page, as opposed to on each of the dat... [09:20:53] good mooooorning equipo A!!! [09:21:03] Mornin' fdans :) [09:23:28] joal: I am back :) [09:23:49] Yay - finishing the review i started, then deploy [09:31:58] Okey, deploying now :) [09:32:52] 10Analytics-Kanban: Final steps to expose project wide unique devices data - https://phabricator.wikimedia.org/T167539#3336133 (10JAllemandou) a:03JAllemandou [09:33:06] 10Analytics-Kanban: Final steps to expose project wide unique devices data - https://phabricator.wikimedia.org/T167539#3336133 (10JAllemandou) a:05JAllemandou>03None [09:36:30] (03PS1) 10Joal: Bump changelog version to 0.0.47 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/358330 [09:36:41] elukey: --^ if you have a minute [09:39:09] (03CR) 10Elukey: [C: 031] Bump changelog version to 0.0.47 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/358330 (owner: 10Joal) [09:39:40] elukey: we can merge [09:41:41] yep we can! [09:41:53] feel free to merge, trivial change [09:42:25] (03CR) 10Joal: [V: 032 C: 032] "self merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/358330 (owner: 10Joal) [10:31:50] joal: all good? Need any help? [10:32:16] elukey: all good, I had just forgotten I had a maven job running, new job started now :) [10:35:28] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3339865 (10elukey) @Marostegui we decided not to proceed with the BBU replacement, the risk it too high with a little gain. We are ok for the moment to use WriteThrough... [10:38:06] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3339869 (10Marostegui) >>! In T166141#3339865, @elukey wrote: > @Marostegui we decided not to proceed with the BBU replacement, the risk it too high with a little gain.... [10:44:03] hellooo [10:45:58] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: Review Megacli Analytics Hadoop workers settings - https://phabricator.wikimedia.org/T166140#3339889 (10elukey) Updated https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [10:46:13] 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: Review Megacli Analytics Hadoop workers settings - https://phabricator.wikimedia.org/T166140#3339890 (10elukey) [10:47:13] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3339893 (10elukey) 05Open>03Resolved [10:48:03] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3286435 (10elukey) @Cmjohnson sorry for the extra pings, we don't need anymore the BBU replacement. Thanks a lot anyway! [10:54:16] elukey: v0,0,47 is only changes in test - I suggest we don't upload it onto refinery [10:54:26] elukey: Do you +1 this ? [10:54:52] !log Refinery-source deployed to archiva [10:54:53] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:55:54] joal: what do you mean with "upload onto refinery" ? [10:56:19] elukey: jars are on archiva, I suggest not copying them in the artifacts folder of refinery [10:56:37] elukey: The patches are on test code only, meaning no actual code change [10:57:28] joal: ah sure, makes sense [10:57:40] elukey: saving some space ? [10:57:47] sorry !, not ? [10:57:51] :) [10:57:58] ok, moving into dpeloyi8ng refinery then [10:58:28] * elukey checks stat1002 [10:58:40] * joal thanks elukey [10:59:06] 1004 seems ok [10:59:47] stat1002 is clean now, you can go :) [10:59:55] thanks elukey [11:01:06] (03CR) 10Joal: [V: 032 C: 032] "self merging for deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/357871 (https://phabricator.wikimedia.org/T167043) (owner: 10Joal) [11:02:18] joal: remember the log message [11:02:31] elukey: I would have had forgotten :) Thanks again [11:03:14] !log Regular weekly deploy of refinery (mostly unique_devices patches) [11:03:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:11:05] !log Deploy refinery onto HDFS [11:11:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:32:12] !log Change production last_access_uniques dataset to unique_devices/per_domain [11:32:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:37:07] joal: ok if I go to lunch? [11:37:14] elukey: please go :) [11:37:46] ooook! [11:37:47] * elukey lunch! [11:59:01] milimetric: good morning! let's talk wikistats tasks in an hour - 3pm CEST? [11:59:40] fdans, yep, planning on it [12:34:28] "You have no pending tasks today!" [12:34:30] * elukey dances [12:34:36] (namely workboard) [12:36:09] oh man, i have done nothing! [12:36:10] today is the day.. [12:36:36] mforns_away: gimme that sweet sweet review: https://gerrit.wikimedia.org/r/#/c/358062/ (when you are not away! :) ) [12:36:58] ottomata: hhhhhhhhhhhhhhhhiIIIIiiiiiiiiiiii [12:37:20] hihihi [12:51:30] elukey: am I missing somethign in namely? I'm on the review page for a person [12:51:32] and I see the questions [12:51:40] but there is no where to enter any text [12:52:36] ottomata: I had the same issue [12:53:14] ottomata: in the "My pending tasks", click the little pen close to the name [12:55:46] ahhh! [12:55:46] ok [12:55:53] thanks [13:02:01] milimetric: cave? [13:02:16] been there for a while [13:02:40] fdans: ^ [13:07:00] 10Analytics-Kanban, 10Analytics-Wikistats: Initial FE code for Wikistats 2.0. Dashboard skeleton - https://phabricator.wikimedia.org/T163814#3340364 (10Milimetric) a:03fdans [13:07:32] 10Analytics-Kanban: Create yaml UI configuration files for Standard Metrics - https://phabricator.wikimedia.org/T166387#3340369 (10Milimetric) [13:07:43] 10Analytics-Kanban: Create yaml UI configuration files for Wikistats metrics - https://phabricator.wikimedia.org/T166388#3294619 (10Milimetric) [13:25:50] (03PS1) 10Joal: Correct bug in unique_devices_per_domain_druid job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/358368 [13:26:10] (03CR) 10Joal: [V: 032 C: 032] "Correcting bug, self merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/358368 (owner: 10Joal) [13:26:22] !log redeploying refinery after bug patch [13:26:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:32:23] 10Analytics-Kanban, 10Patch-For-Review: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3340421 (10elukey) I came up with a preliminary list of ALTER statements to run: https://phabricator.wikimedia.org/P5570 Procedure: 1) `SELECT TABLE_NAME, COLUMN_NAME, DATA_TY... [13:36:48] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching): Expose revision-create in EventStreams - https://phabricator.wikimedia.org/T167670#3340424 (10Ottomata) [13:37:02] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching): Expose revision-create in EventStreams - https://phabricator.wikimedia.org/T167670#3340441 (10Ottomata) Also related: T167180 [13:37:23] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching), 10User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#2761082 (10Ottomata) [13:37:25] 10Analytics, 10EventBus, 10ORES, 10Reading-Infrastructure-Team-Backlog, and 3 others: Emit revision-score event to EventBus and expose in EventStreams - https://phabricator.wikimedia.org/T167180#3319947 (10Ottomata) [13:46:23] fdans: ready when you are [13:47:31] omw [13:55:33] 10Analytics-Kanban: Routing - https://phabricator.wikimedia.org/T167672#3340529 (10Milimetric) [13:57:19] 10Analytics-Kanban: Address design feedback from Volker - https://phabricator.wikimedia.org/T167673#3340552 (10Milimetric) [13:57:38] 10Analytics-Kanban, 10Patch-For-Review: Rename last_access_uniques to per-domain uniques - https://phabricator.wikimedia.org/T167043#3340565 (10JAllemandou) [13:58:26] Taking a break a-team, unique_devices_per_domain are up and running, still some cleaning and doc to do this evening, as well as starting the project_wide [14:01:02] 10Analytics-Kanban: Address design feedback from Volker - https://phabricator.wikimedia.org/T167673#3340570 (10Milimetric) p:05Triage>03Normal [14:03:24] 10Analytics-Kanban: Re-read Round 2 feedback on mediawiki and make any critical items into tasks - https://phabricator.wikimedia.org/T167674#3340573 (10Milimetric) [14:05:20] 10Analytics-Kanban: Implement Wiki Selector Widget - https://phabricator.wikimedia.org/T167675#3340588 (10Milimetric) [14:06:27] 10Analytics-Kanban: Implement Topic Selector Widget - https://phabricator.wikimedia.org/T167676#3340600 (10Milimetric) [14:08:25] 10Analytics-Kanban: Build Dashboard on top of dynamic data - https://phabricator.wikimedia.org/T167677#3340613 (10Milimetric) [14:09:34] o/ ottomata [14:09:56] I've got some volunteers who might be interested in getting kafka/eventbus --> changeprop running in labs. [14:11:24] 10Analytics-Kanban: Data Model using Crossfilter, called DimensionalData - https://phabricator.wikimedia.org/T167678#3340638 (10Milimetric) [14:12:13] 10Analytics-Kanban: Interface from Graphs to DimensionalData - https://phabricator.wikimedia.org/T167679#3340653 (10Milimetric) [14:12:36] ah awesome halfak [14:12:43] 10Analytics-Kanban: Interface from Detail page to DimensionalData - https://phabricator.wikimedia.org/T167680#3340666 (10Milimetric) [14:12:52] Would you be interested in helping them out from the prod-side of things? [14:12:58] don't know too much about change prop, but yes totally [14:13:06] have they done it in mw-vagrant? its super easy there [14:13:26] Cool. I'll drag them over to this channel once they are online ^_^ [14:13:30] k cool [14:13:59] 10Analytics-Kanban: AQS Api works with DimensionalData - https://phabricator.wikimedia.org/T167681#3340679 (10Milimetric) [14:13:59] ottomata, yeah. I think so. I found 'em because they are working on getting a icinga-for-labs. So I brought up the idea of getting other prod services running in labs. [14:17:32] 10Analytics-Kanban: AQS Api works with DimensionalData - https://phabricator.wikimedia.org/T167681#3340698 (10Milimetric) p:05Triage>03Normal [14:17:56] 10Analytics-Kanban: Re-read Round 2 feedback on mediawiki and make any critical items into tasks - https://phabricator.wikimedia.org/T167674#3340701 (10Milimetric) p:05Triage>03Normal [14:18:23] 10Analytics-Kanban: Implement Wiki Selector Widget - https://phabricator.wikimedia.org/T167675#3340702 (10Milimetric) p:05Triage>03Normal [14:18:30] 10Analytics-Kanban: Re-read Round 2 feedback on mediawiki and make any critical items into tasks - https://phabricator.wikimedia.org/T167674#3340573 (10Milimetric) [14:20:19] 10Analytics-Kanban: Implement Topic Selector Widget - https://phabricator.wikimedia.org/T167676#3340718 (10Milimetric) p:05Triage>03Normal [14:27:58] 10Analytics-Kanban: Build Dashboard on top of dynamic data - https://phabricator.wikimedia.org/T167677#3340742 (10Milimetric) p:05Triage>03Normal [14:28:21] 10Analytics-Kanban: Data Model using Crossfilter, called DimensionalData - https://phabricator.wikimedia.org/T167678#3340745 (10Milimetric) p:05Triage>03Normal [14:28:38] 10Analytics-Kanban: Interface from Graphs to DimensionalData - https://phabricator.wikimedia.org/T167679#3340747 (10Milimetric) p:05Triage>03Normal [14:28:58] 10Analytics-Kanban: Interface from Detail page to DimensionalData - https://phabricator.wikimedia.org/T167680#3340749 (10Milimetric) p:05Triage>03Normal [14:30:15] 10Analytics-Kanban, 10Analytics-Wikistats: Implement pageviews and unique devices detail pages in Wikistats UI - https://phabricator.wikimedia.org/T163817#3211246 (10Milimetric) [14:32:32] 10Analytics-Kanban: AQS Api works with DimensionalData - https://phabricator.wikimedia.org/T167681#3340759 (10Milimetric) [14:32:34] 10Analytics-Kanban: Interface from Detail page to DimensionalData - https://phabricator.wikimedia.org/T167680#3340760 (10Milimetric) [14:32:36] 10Analytics-Kanban: Interface from Graphs to DimensionalData - https://phabricator.wikimedia.org/T167679#3340761 (10Milimetric) [14:32:38] 10Analytics-Kanban: Implement Wiki Selector Widget - https://phabricator.wikimedia.org/T167675#3340765 (10Milimetric) [14:32:40] 10Analytics-Kanban: Implement Topic Selector Widget - https://phabricator.wikimedia.org/T167676#3340764 (10Milimetric) [14:32:42] 10Analytics-Kanban: Re-read Round 2 feedback on mediawiki and make any critical items into tasks - https://phabricator.wikimedia.org/T167674#3340766 (10Milimetric) [14:32:44] 10Analytics-Kanban: Build Dashboard on top of dynamic data - https://phabricator.wikimedia.org/T167677#3340763 (10Milimetric) [14:32:46] 10Analytics-Kanban: Address design feedback from Volker - https://phabricator.wikimedia.org/T167673#3340767 (10Milimetric) [14:32:48] 10Analytics-Kanban, 10Analytics-Wikistats: Implement pageviews and unique devices detail pages in Wikistats UI - https://phabricator.wikimedia.org/T163817#3340758 (10Milimetric) [14:36:42] 10Analytics-Kanban: Deploy new Wikistats to stats.wikimedia.org/v2 - https://phabricator.wikimedia.org/T167684#3340773 (10Milimetric) [14:36:55] 10Analytics-Kanban: Deploy new Wikistats to stats.wikimedia.org/v2 - https://phabricator.wikimedia.org/T167684#3340785 (10Milimetric) p:05Triage>03Normal [14:37:16] 10Analytics-Kanban: Deploy new Wikistats to stats.wikimedia.org/v2 - https://phabricator.wikimedia.org/T167684#3340773 (10Milimetric) [14:37:18] 10Analytics-Kanban, 10Analytics-Wikistats: Implement pageviews and unique devices detail pages in Wikistats UI - https://phabricator.wikimedia.org/T163817#3340787 (10Milimetric) [14:40:30] o/ paladox [14:40:35] ottomata, meet paladox [14:40:35] :) [14:40:53] paladox has been maintaining icinga for labs. I've been talking to him about EventBus/ChangeProp [14:40:58] ottomata is EventBus [14:40:59] :) [14:41:04] yep [14:41:13] hiiiyyaaa [14:41:18] hi [14:46:38] mforns: helloooooo [14:46:43] hey elukey ! [14:46:49] how are you doing? [14:46:54] good :] [14:46:56] you? [14:48:27] good, finally namely looks clean and I am happier :P [14:48:44] mforns: did you see the alter tables that I put in the task? [14:48:52] elukey, not yey [14:48:54] Manuel was not happy about them :P [14:48:55] *yet [14:48:59] mmmm [14:49:02] will look [14:49:31] elukey, doing peer reviews now though [14:49:57] mforns: sure sure [14:52:34] elukey, where is the task? I can not find it :[ [14:54:26] mforns: https://phabricator.wikimedia.org/T167162 [15:01:04] ping elukey ottomata [15:02:31] bad ops are bad [15:02:34] sorryyy [15:03:11] AHHH [15:03:16] sorry was chatting in mwsec [15:24:51] 10Analytics, 10Pageviews-API, 10ProofreadPage: API: image thumb-url for ProofreadPages - https://phabricator.wikimedia.org/T167200#3320626 (10Nuria) @Mpaa: not sure who is recipient of this task but we do not think is analytics, removing pageview api that deals with pageviews (not files) [15:26:46] 10Analytics-Dashiki, 10Analytics-Kanban, 10MW-1.30-release-notes (WMF-deploy-2017-06-13_(1.30.0-wmf.5)), 10Patch-For-Review, 10Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3341008 (10... [15:28:30] 10Analytics, 10Analytics-Dashiki, 10Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3341012 (10dr0ptp4kt) [15:28:53] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3323476 (10Nuria) Seems this this data is pretty clean on database, is the benefit of scoping this being able to look data across projects more easily? [15:29:08] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3341017 (10Nuria) p:05Triage>03Low [15:29:17] 10Analytics-Kanban, 10Patch-For-Review: Load webrequest raw data into druid so ops can use it for troubleshooting - https://phabricator.wikimedia.org/T166967#3313173 (10fgiunchedi) thanks @JAllemandou ! I give it a quick try and it looks very interesting, how often is the data loaded from webrequest? IOW how... [15:31:38] 10Analytics, 10Analytics-EventLogging: Form for editing schema On eventlogging - https://phabricator.wikimedia.org/T49593#3341026 (10Nuria) [15:32:02] 10Analytics-Kanban, 10Patch-For-Review: Load webrequest raw data into druid so ops can use it for troubleshooting - https://phabricator.wikimedia.org/T166967#3341028 (10JAllemandou) Hey @fgiunchedi, you're welcome, I'm glad it's useful. Data is loaded hourly, with between 1h30 and 2h lag when everything works... [15:33:02] 10Analytics, 10Analytics-EventLogging: Form for editing schema On eventlogging - https://phabricator.wikimedia.org/T49593#517835 (10Nuria) The schemas in eventlogging can be edited with json editor. [15:33:09] 10Analytics, 10Analytics-EventLogging: Form for editing schema On eventlogging - https://phabricator.wikimedia.org/T49593#3341032 (10Nuria) 05Open>03declined [15:34:03] 10Analytics, 10Analytics-EventLogging, 10Puppet: Eventlogging file logging code split weirdly between role and base class - https://phabricator.wikimedia.org/T86745#975424 (10Nuria) Old task, not relevant. [15:34:10] 10Analytics, 10Analytics-EventLogging, 10Puppet: Eventlogging file logging code split weirdly between role and base class - https://phabricator.wikimedia.org/T86745#3341037 (10Nuria) 05Open>03Resolved [15:36:20] 10Analytics: Add 'page_is_redirect' field to the mediawiki_history Data Lake tables - https://phabricator.wikimedia.org/T167396#3331684 (10Nuria) FYI that there is no way to do this historically without parsing content (new events could come via EventBus) [15:37:26] 10Analytics: Provide historical redirect flag in Data Lake edit data - https://phabricator.wikimedia.org/T161146#3341058 (10Nuria) [15:38:16] 10Analytics-Kanban: Implement purging settings for Schema:ReadingDepth - https://phabricator.wikimedia.org/T167439#3341061 (10Nuria) [15:38:46] 10Analytics, 10Analytics-Kanban, 10Page-Previews, 10Reading-Web-Backlog: Update purging settings for Schema:Popups - https://phabricator.wikimedia.org/T167449#3341063 (10Nuria) [15:39:02] 10Analytics-Kanban, 10Page-Previews, 10Reading-Web-Backlog: Update purging settings for Schema:Popups - https://phabricator.wikimedia.org/T167449#3333176 (10Nuria) [15:43:28] 10Analytics: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3334762 (10Nuria) We will to update a bit our ingestion schema to do this, however. [15:44:56] 10Analytics: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3341089 (10Nuria) >Other dashboards would benefit from the same treatment, for example in Pageviews it would also be quite convenient to have browser major/minor versions as integers, again to be able to filter by ver... [15:45:12] 10Analytics: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3334762 (10Nuria) p:05Triage>03Normal [15:46:55] 10Analytics: Add caused_by_user_text to mediawiki_page_history - https://phabricator.wikimedia.org/T167608#3338630 (10Nuria) Is this about listing the IP in the revision create events that have it? [15:50:20] 10Analytics: Add caused_by_user_text to mediawiki_page_history - https://phabricator.wikimedia.org/T167608#3341126 (10Nuria) p:05Triage>03Low [15:52:15] 10Analytics: Add caused_by_user_text to mediawiki_page_history - https://phabricator.wikimedia.org/T167608#3341132 (10Milimetric) [15:52:17] 10Analytics: Meta-statistics on MediaWiki history reconstruction process - https://phabricator.wikimedia.org/T155507#3341131 (10Milimetric) [15:57:13] 10Analytics-Kanban, 10Patch-For-Review: Update per-domain uniques fresh-sessions computation - https://phabricator.wikimedia.org/T167005#3341148 (10JAllemandou) [16:00:12] 10Analytics, 10MediaWiki-API: Copy cached API requests from raw webrequests table to ApiAction - https://phabricator.wikimedia.org/T155478#3341159 (10Nuria) This ticket has several requests, regarding of being able to harvest API cached requests: @Tgr: I think part of this work can be addressed with tagging... [16:02:28] 10Analytics, 10Pageviews-API: Create fake data for beta AQS deployment - https://phabricator.wikimedia.org/T120841#3341165 (10Nuria) We have loaded several datasets on AQS beta: https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS#Load_data_into_beta [16:02:34] 10Analytics: Making tests environment for pageview API deployments - https://phabricator.wikimedia.org/T131773#3341167 (10Nuria) [16:02:36] 10Analytics, 10Pageviews-API: Create fake data for beta AQS deployment - https://phabricator.wikimedia.org/T120841#3341166 (10Nuria) 05Open>03Resolved [16:02:38] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services, 10scap2, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#3341168 (10Nuria) [16:03:19] 10Analytics, 10Analytics-Dashiki: Dashiki (layout fixes) mostly mobile - https://phabricator.wikimedia.org/T147007#2677640 (10Nuria) 05Open>03declined [16:04:05] 10Analytics, 10Fundraising-Analysis, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#3341171 (10Nuria) 05Open>03Resolved [16:05:03] 10Analytics, 10Fundraising-Analysis, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Provide performant query access to banner show/hide numbers - https://phabricator.wikimedia.org/T90649#1064105 (10Nuria) If newer banner dataset on druid needs more fields please file a task a to that extent [16:06:44] 10Analytics, 10Analytics-Cluster, 10Analytics-Wikistats: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1066753 (10Nuria) [16:07:16] 10Analytics-Kanban, 10Analytics-Wikistats: Re-read Round 2 feedback on mediawiki and make any critical items into tasks - https://phabricator.wikimedia.org/T167674#3341206 (10Milimetric) [16:07:26] 10Analytics-Kanban, 10Analytics-Wikistats: Data Model using Crossfilter, called DimensionalData - https://phabricator.wikimedia.org/T167678#3341207 (10Milimetric) [16:07:35] 10Analytics-Kanban, 10Analytics-Wikistats: Routing - https://phabricator.wikimedia.org/T167672#3341208 (10Milimetric) [16:07:37] 10Analytics, 10Analytics-Cluster, 10Analytics-Wikistats: Create Daily & Monthly pageview dump with country data - https://phabricator.wikimedia.org/T90759#1066753 (10Nuria) [16:07:39] 10Analytics-Kanban: Wikistats 2.0. - https://phabricator.wikimedia.org/T130256#3341209 (10Nuria) [16:07:41] 10Analytics-Kanban, 10Analytics-Wikistats: AQS Api works with DimensionalData - https://phabricator.wikimedia.org/T167681#3341211 (10Milimetric) [16:07:47] 10Analytics-Kanban, 10Analytics-Wikistats: Interface from Detail page to DimensionalData - https://phabricator.wikimedia.org/T167680#3341212 (10Milimetric) [16:07:54] 10Analytics-Kanban, 10Analytics-Wikistats: Interface from Graphs to DimensionalData - https://phabricator.wikimedia.org/T167679#3341214 (10Milimetric) [16:08:01] 10Analytics-Kanban, 10Analytics-Wikistats: Build Dashboard on top of dynamic data - https://phabricator.wikimedia.org/T167677#3341215 (10Milimetric) [16:08:23] 10Analytics-Kanban, 10Analytics-Wikistats: Implement Wiki Selector Widget - https://phabricator.wikimedia.org/T167675#3341216 (10Milimetric) [16:08:30] 10Analytics-Kanban, 10Analytics-Wikistats: Implement Topic Selector Widget - https://phabricator.wikimedia.org/T167676#3341217 (10Milimetric) [16:08:39] 10Analytics-Kanban, 10Analytics-Wikistats: Address design feedback from Volker - https://phabricator.wikimedia.org/T167673#3341218 (10Milimetric) [16:08:48] 10Analytics-Kanban, 10Analytics-Wikistats: Deploy new Wikistats to stats.wikimedia.org/v2 - https://phabricator.wikimedia.org/T167684#3341219 (10Milimetric) [16:15:29] 10Analytics: Make a script to automatise the 4 commands to run for aqs deployment - https://phabricator.wikimedia.org/T133863#2247732 (10Nuria) Docs are now a lot better, let's try to keep an eya if we run inti issues still when we follow them: https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS#Deployment [16:15:35] 10Analytics: Make a script to automatise the 4 commands to run for aqs deployment - https://phabricator.wikimedia.org/T133863#3341251 (10Nuria) 05Open>03declined [16:20:27] 10Analytics-Kanban, 10Patch-For-Review: Load webrequest raw data into druid so ops can use it for troubleshooting - https://phabricator.wikimedia.org/T166967#3341263 (10Nuria) Let's go ahead with tehse changes then, if there not additional suggestions. [16:39:56] joal: pivot restarted with your last change [16:41:29] elukey: that's weird, I can't see the name change :( [16:46:15] joal: because I restarted pivot before running puppet! Just wanted to check if you were paying attention [16:46:21] * elukey restarts pivot again [16:46:37] * joal b [16:46:53] * joal blames himself for not paying attention enough when submitting patches [16:47:22] joal: better now? [16:47:32] indeed :) [16:47:37] many thanks again ! [16:47:40] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Services (watching): Expose revision-create in EventStreams - https://phabricator.wikimedia.org/T167670#3341405 (10Halfak) [16:48:33] :) [16:52:25] nuria_: I updated the docs (except AQS, which endpoint has not yet changed) to the new names. [16:59:11] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services, 10scap2, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#3341469 (10greg) Status on this old task? :) [17:11:51] (03PS1) 10Joal: Correct bug in unique_devices_project_wide_monthly [analytics/refinery] - 10https://gerrit.wikimedia.org/r/358403 [17:12:19] (03CR) 10Joal: [V: 032 C: 032] "Self merging bug for deploy." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/358403 (owner: 10Joal) [17:12:31] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3341548 (10GoranSMilovanovic) p:05Low>03High @Nuria: I am currently developing an application that should provide a thorough insight into Wikidata usage across the client projects. The prototype... [17:12:38] elukey, I've looked into the alter tables thing [17:13:40] elukey: I think I need another minute of your time for a stat1002 cleanup before last deploy of today [17:14:06] As far as I understand, while we're not adding UNIQUE or changing TYPE or touching indexes or primary/foreign keys, we do not need to iterate over all records [17:14:44] and as we change from NOT NULL to NULL, there are no not-null checks as well [17:14:48] so I think we're fine [17:15:17] except maybe... the wiki field has an index on it in all tables, shared with timestamp: (wiki, timestamp) [17:15:32] and also, there are special tables, like Edit, that have other indexes [17:15:57] not sure though if changing to NULLABLE makes mariadb recompute the indexes...? [17:16:22] I don't think it will take a huge amount of time too [17:16:24] joal: sure [17:16:55] elukey, will comment on task pinging marostegui [17:17:30] mforns: no need, he is not saying that it will take a huge amount of time.. only that those are a lot of alters and it might take a bit of time [17:17:33] but he is ok with them [17:17:42] elukey, I see [17:17:45] the only thing that I am wondering is the syntax [17:19:59] elukey, we could merge all columns of the same table in a single statement [17:20:33] joal: deleted one scap rev and freed ~4GB, should be enough [17:20:51] elukey: Thanks ! [17:21:08] mforns: not sure if it will make a big difference, plus it would mess up a lot the query no? [17:21:10] ALTER TABLE blah_123 MODIFY col_1 type_1 NULL, MODIFY col_2 type_2 NULL, ... MODIFY col_N type_N NULL; [17:21:16] !log Last deploy of the day for uniques patch [17:21:17] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:21:29] but yeah we could [17:21:40] I tried to keep the alters as self contained as possible [17:22:24] going afk people, will read later on for any pings! [17:22:30] * elukey afk! [17:23:01] Bye elukey [17:23:03] 10Analytics: Some fields in Pivot should be numbers - https://phabricator.wikimedia.org/T167494#3341592 (10JAllemandou) @Gilles: Currently changing the schema to have, as you suggested, `time_firstbyte: float` and `response_size: Long`. Do you think it would be valuable to have `http_response: Long`? I'm assumin... [17:23:49] elukey, bye! [17:26:09] 10Analytics, 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3341608 (10GoranSMilovanovic) [17:27:33] 10Analytics-Kanban, 10Patch-For-Review: Make non-nullable columns in EL database nullable - https://phabricator.wikimedia.org/T167162#3341610 (10mforns) I think the idea and script are pretty good! If we want to reduce the number of alter statements, we could join all statements that pertain to the same table... [17:34:05] joal: are there any near plans to update the version of spark? or should I target 1.6? [17:34:48] schana: hiiii [17:34:56] hi ottomata [17:35:02] we would love to support that cluster wide, but it would be much less work for us if we wait for cloudera to release [17:35:21] i did a little bit of work on that a few months ago, but we put it aside because it was a little complicated to do it well. [17:35:36] but! (if you don't already know) you can run whatever version of spark you want on the cluster [17:35:51] (I don't know anything about running spark on the cluster) [17:35:53] yet [17:35:53] we can't productionize stuff with it, but if you are running your own stuff then you can def use it [17:36:28] this is being written with the eventual intent of being productionized [17:36:57] aye ok [17:37:49] I don't think there are any features I'm super dependent on, but I'll have to go back through and redo the mllib stuff, I think [17:38:15] you started wtih 2.0? [17:38:19] * schana should have checked the version before writing the code [17:38:20] 2.1 [17:40:49] schana: will you comment here with a description of your use case? https://phabricator.wikimedia.org/T158334 [17:40:52] might help us prioritize [17:41:26] sure, ottomata [17:41:37] thanks [17:44:38] 10Analytics: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3341742 (10Ottomata) Bumped this cloudera forum thread: https://community.cloudera.com/t5/Beta-Releases-RecordService/Spark-2-1-official-beta/m-p/49476 [17:48:26] 10Analytics: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3033748 (10schana) We have [[ https://github.com/schana/recommendation-translation | work ongoing ]] in T162912 that was initially built against 2.1 with the eventual intent of productization. It uses... [17:50:21] 10Analytics, 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3341596 (10Reedy) > Intensively searched for similar problem reports and solutions; many users are complaining about this; still with no success on my Labs instance... [18:00:25] 10Analytics, 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3341832 (10GoranSMilovanovic) @Reedy "Complaining where? I presume you don't mean on labs" - No, I don't mean complaining on Labs; I mean: many RStudio Shiny Server... [18:10:34] 10Analytics, 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3341596 (10bd808) If all you need is a transparent reverse proxy, you should be able to open port 3838 in your project's security groups and then target the wdcm.wm... [18:29:08] 10Analytics, 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3341941 (10GoranSMilovanovic) @bd808 @Reedy I already have a security group for Shiny Server, port 3838 opened. My /etc/nginx/nginx.conf is as follows (*exactly*... [18:35:33] milimetric: meeting? [18:40:53] kaldari: sorry, something came up, one more minute [19:02:34] milimetric: http://tools.wmflabs.org/?list [19:07:17] 10Analytics-Kanban, 10Patch-For-Review: Count project-wide unique devices (like *.wikipedia.org) - https://phabricator.wikimedia.org/T143928#3342281 (10JAllemandou) [19:29:45] (03CR) 10Nuria: Add sampled-webrequest druid loading oozie jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) (owner: 10Joal) [19:30:36] (03CR) 10Joal: "We do. Will add to the patch (waiting for an answer from Gilles)." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) (owner: 10Joal) [19:32:21] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services, 10scap2, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#3342395 (10Nuria) An instance of aqs on beta for testing for analytics already exists. This instance is a testing environment helpful for analytics changes but it do... [19:38:41] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342421 (10Nuria) @GoranSMilovanovic, sorry but this is not work we can tackle in the next quarter, we need to do several upgrades to the current mediawiki edit reconstruction that have higher priori... [19:40:25] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342423 (10Halfak) Wow! I didn't realize this would be that much work. @Ottomata thought it would be a little bit of configuration around sqoop. [19:42:46] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342442 (10Nuria) Let's see if thisi data is on labs, that might make things easier. [19:44:00] ottomata: you had no issue with merging mforns CR, right? https://gerrit.wikimedia.org/r/#/c/357665/ [19:44:20] ya +1 on that [19:44:25] merge away [19:46:20] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342453 (10Halfak) Oh! It is. The table exists without filtering of any type in labs. [19:48:43] 10Analytics, 10Analytics-EventLogging, 10Composer, 10Easy: EventLogging has Invalid composer.json - https://phabricator.wikimedia.org/T167309#3342465 (10Legoktm) 05Open>03declined This is intentional - it's not supposed to be published on packagist, nor installable via composer. [19:50:26] ottomata: just did, test pass [19:51:30] milimetric: , yt? [19:51:34] hey ottomata [19:51:44] i'm trying to look to see if that wikidata table is in labsdb [19:51:46] for sqooping [19:51:48] but am failing [19:51:54] i'm using the same creds that the sqoop script uses [19:51:57] but i can't see any dbs [19:52:51] labsdb? [19:52:59] I don't think we can sqoop from labsdb... [19:53:15] oh wait... we do sqoop from the new labsdb [19:53:29] you do atm but it's not all shards still [19:53:31] iiuc [19:53:31] milimetric: right, new replicas [19:53:50] ottomata: so how are you connecting to labsdb? [19:53:57] where's this sqoop script [19:54:10] mysql -hlabsdb-analytics.eqiad.wmnet -uYYYYY -pXXXXX [19:54:19] milimetric: didn't you write it? [19:54:19] haha [19:54:20] refinery [19:54:25] bin/mediawiki-sqoop-tables [19:54:30] sorry [19:54:31] no, that was jo [19:54:33] sqoop-mediawiki-tables [19:54:34] ah [19:54:59] hm, maybe that user doesn't have rights to list dbs? [19:55:01] one sec [19:55:08] AH [19:55:09] -P [19:55:11] _p [19:55:12] got it [19:55:34] ottomata: indeed - in labs, dbs are postfized (they actually are views I think [19:55:37] wikidatawiki_p? [19:55:46] that makes sense [19:55:49] enwiki_p.wbc_entity_usage [19:56:11] ottomata: while at it, can you grab some size estimates? [19:56:30] like # recods? [19:56:32] 10Analytics, 10Analytics-EventLogging, 10Composer, 10Easy: EventLogging has Invalid composer.json - https://phabricator.wikimedia.org/T167309#3342572 (10dbarratt) >>! In T167309#3342465, @Legoktm wrote: > This is intentional - it's not supposed to be published on packagist, nor installable via composer. W... [19:56:51] ottomata: yes, and actual size on disk (if available) [19:57:25] 10Analytics, 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3342573 (10bd808) >>! In T167702#3341941, @GoranSMilovanovic wrote: > @bd808 @Reedy > > I already have a security group for Shiny Server, port 3838 opened. This... [19:57:59] joal: 13990131 in enwiki_p [19:59:03] enwiki [19:59:08] 1.2G Jun 12 19:58 wbc_entity_usage.ibd [20:00:58] thanks ottomata [20:01:22] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342601 (10Ottomata) Just checked, and the table does exist in the new labsdb replicas we are using for sqooping. I've heard that these replicas don't yet have all DB shards though, so some wikis mi... [20:01:43] small rows, not that many - shouldn't be difficult not really slow down the sqooping process [20:03:05] yeah its really just a join table [20:04:29] ottomata: This could be a good exercise to have a more generic way to sqoop tables (less code, more conf)? [20:04:53] ottomata, just an fyi, enwiki is not the biggest consumer of wbc_entity_usage [20:05:00] Probably ruwiki has more records. [20:05:05] But won't be a huge difference. [20:05:19] ottomata: careful, halfak has eyes behing his back ;) [20:05:24] :D [20:05:40] * halfak gets back to writing reviews of all the awesome people he works with [20:05:47] hehe, joal maybe so, but we gotta be careful or nuria won't let us do it [20:06:02] she'll only let us do it if we do it really quick without her realizing it :p [20:06:07] ottomata: yup, more pressing things on the way [20:06:47] lol [20:10:18] gotta run a little early today yall ttyl! [20:11:03] bye ott [20:11:12] Arf - gone already [20:14:11] ebernhar|lunch: Hola! are these the extent of changes that you added on top of eventlogging to be able to do better AB testing? https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/master/modules/ext.wikimediaEvents.searchSatisfaction.js [20:19:53] Gone for tonight a-team - see you tomorrow (late start for me) [20:19:59] byyye joal [20:20:46] joal, halfak : we can look at doing those changes once our tagging code is on its way and once we have added edit count to data lake, does that make sense? [20:21:11] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342705 (10Nuria) We can look at doing those changes once our tagging code is on its way and once we have added edit count to data lake, does that make sense? [20:22:34] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342710 (10GoranSMilovanovic) @Ottomata I can live with "some wikis might be missing " in the beginning. Thanks. [20:23:08] 10Analytics: Sqoop wbc_entity_usage from all wikis into hadoop (HDFS) - https://phabricator.wikimedia.org/T167290#3342711 (10Nuria) p:05High>03Normal [20:40:15] nuria_: yes, everything we've added is in that one file [20:43:58] at a high level, a few pieces: SessionState to keep a browser in a test and time them out after some specified time (and remember a little bit of data). Some code for triggering events based on 'visible' time, rather than how long page has been open. Some handling for click events on non-sendbeacon enabled browsers (should perhaps consider disabling AB tests on those instead), and some code for detecting where a click came from, wh [21:08:18] ebernhardson: super thanks [21:22:26] 10Analytics, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Data request for logs from SparQL interface at query.wikidata.org - https://phabricator.wikimedia.org/T143819#3342917 (10Smalyshev) [22:34:42] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 5 others: Record an EventLogging event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3343163 (10Nuria) ping @kaldari about a bit of metric definition