[05:09:51] (03PS1) 10Catrope: Add stats for the "New wikitext mode" beta feature [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/343247 [09:05:45] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3109196 (10Marostegui) And it is not only being recreated, but used as of today: ``` root@EVENTLOGGING m4[log]> select t... [09:22:44] 06Analytics-Kanban, 10DBA, 13Patch-For-Review: Change length of userAgent column on EL tables - https://phabricator.wikimedia.org/T160454#3109219 (10Marostegui) Thanks for the list of tables. From the DBA side, this would be the only thing to execute really (assuming you just want to add the "_ 15423246" to... [10:32:35] hey team :] [10:32:47] Hi mforns [10:33:22] o/ [10:51:14] !log restarted mediacounts-archive-wf-2017-03-16 [10:51:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:08:10] (03PS8) 10Joal: Add oozie job loading MW history in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/328154 (https://phabricator.wikimedia.org/T141473) [11:29:26] (03PS18) 10Joal: Port standard metrics to reconstructed history [analytics/refinery] - 10https://gerrit.wikimedia.org/r/322103 (https://phabricator.wikimedia.org/T160155) (owner: 10Milimetric) [11:30:20] hello people, reimaging an1044 to debian :) [11:30:29] Hi elukey ! Please please :) [11:30:43] * joal feels debizaner today :) [11:30:57] :) [11:31:05] it is a loooong way though [11:31:11] this is the 5th host :P [11:31:16] so booooringggg [11:31:45] elukey: I'm very gratefull you it nonetheless :) [11:52:30] 10Analytics, 10Analytics-Cluster, 06Operations, 13Patch-For-Review, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3109448 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analy... [12:07:39] (03PS13) 10Joal: Add oozie jobs for mw history denormalized [analytics/refinery] - 10https://gerrit.wikimedia.org/r/341030 (https://phabricator.wikimedia.org/T160074) [12:18:03] 10Analytics, 10Analytics-Cluster, 06Operations, 13Patch-For-Review, 15User-Elukey: Reimage all the Hadoop worker nodes to Debian Jessie - https://phabricator.wikimedia.org/T160333#3109495 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1044.eqiad.wmnet'] ``` and were **ALL** succe... [12:25:29] * elukey restarts jobs. [12:26:16] joal: is mediawiki-history-denormalize your job? [12:26:33] elukey: it is [12:26:39] sorry :( [12:26:53] elukey: excuse me, forgot to change email address :S [12:27:03] !log restarted webrequest-load-wf-text-2017-3-17-10 [12:27:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:37:10] * elukey lunch! [13:15:47] taking a break a-team [13:15:54] o/ [13:16:02] hey joal, if you have a spare moment, could you point me in the right direction on https://phabricator.wikimedia.org/T159706 ? tl;dr I'm having performance issues with a query relying on multikey indexing in mongodb and want to know if (a) this datastore makes sense or (b) what better methods exist [13:16:09] joal: no rush [13:16:11] (03PS4) 10Mforns: Add oozie workflow to load projectcounts to AQS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/339421 (https://phabricator.wikimedia.org/T156388) [13:16:27] schana: short answer b [13:16:57] schana: short answer before I leave: I advise not to use mongodb [13:17:17] schana: previous experience with this datastore was not good from an operational side [13:17:28] schana: leaving now - we'll take again later [13:17:36] thanks [13:19:59] (03PS5) 10Mforns: Add oozie workflow to load projectcounts to AQS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/339421 (https://phabricator.wikimedia.org/T156388) [13:21:19] schana: i concur with joseph, even though i have little experience with mongodb. what i've mostly heard is not great: https://jepsen.io/analyses/mongodb-3-4-0-rc3 [13:38:43] 06Analytics-Kanban, 10Wikimedia-Stream: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus - https://phabricator.wikimedia.org/T160748#3109641 (10Ottomata) [13:38:58] 06Analytics-Kanban, 10Wikimedia-Stream: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus - https://phabricator.wikimedia.org/T160748#3109654 (10Ottomata) p:05Triage>03Normal [13:39:05] 06Analytics-Kanban, 10Wikimedia-Stream: Create /v2/schema/:schema_uri endpoint for eventstreams that proxies schemas from eventbus - https://phabricator.wikimedia.org/T160748#3109641 (10Ottomata) [13:41:56] ottomata: HHHiiiiiiiiiiiiiiiiiiiiiiiiii [13:42:05] how is the weather today in NYC? [13:42:29] HIIII [13:42:36] slush on the ground, but pretty sunny [13:42:42] how's it out there elukey? [13:43:02] very sunny, this morning it was lovely [14:19:01] mforns: can you remind me, what's the beginning of time in legacy pageviews? [14:19:24] fdans, Dec 18 2007 [14:19:33] wonderful! [14:19:38] or Dec 12? [14:19:40] one sec [14:20:03] :] [14:20:05] dec 12 was my beginning of time in wmf :P [14:20:13] hehe [14:20:14] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3109759 (10Halfak) OK. Time to try to ping the larger set of people who have databases here. Here's the databases that match Phab users: * @dartar (dartar) * @drdee (diederi... [14:22:13] fdans, Dec 9th 2007 at 18h [14:22:27] awesome, thank you marcel :) [14:22:33] np :] [14:24:03] !log analytics1044 back in the cluster [14:24:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:32:37] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3109809 (10Ottomata) Wow, old DB! decleramaul, nimish and rfaulk we can certainly get rid of. [14:47:46] elukey: o/ [14:48:36] :) [14:48:40] just pinged you on ops [14:48:46] ha! [14:48:51] tag, you're it! [14:51:11] (03PS6) 10Mforns: Add oozie workflow to load projectcounts to AQS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/339421 (https://phabricator.wikimedia.org/T156388) [15:00:20] fdans: standduppp [15:12:11] 06Analytics-Kanban, 10Analytics-Wikistats: Visual Language for http://stats.wikimedia.org replacement - https://phabricator.wikimedia.org/T152033#2836084 (10ashgrigas) {F6669373} [15:56:53] 10Analytics, 10Analytics-EventLogging, 10DBA, 10ImageMetrics: Drop EventLogging tables for ImageMetricsLoadingTime and ImageMetricsCorsSupport - https://phabricator.wikimedia.org/T141407#3110057 (10Nuria) ping @Jdforrester-WMF looks like you need to remove the instrumentation that is sending events. [16:08:41] nuria, when is the wikistats closing meeting happening? I don't have an invite :( [16:09:08] mforns: we did not have one, sorry, we just had a 1 on 1 ash and myself [16:09:14] oh ok [16:09:27] np thx! :] [16:15:06] is kafka on 0.9 now? i see 0.8.2 in the wikimedia repository, but there was an email about 0.9 way back in may [16:54:00] ebernhardson: yep we are on 0.9 for both eventbus and analytics clusters [16:55:01] milimetric: sqoop is driving me mad :( [16:58:47] * elukey off for the weekend, bye team! [17:00:03] joal: This order has shipped. The delivery date is projected to be 03/24/17. [17:00:08] MOAR Hadoop nodes [17:00:09] :D [17:00:17] YAY ! Thanks for the update elukey :) [17:10:47] 10Analytics-Tech-community-metrics: Updated data in mediawiki-identities DB not deployed onto wikimedia.biterg.io? - https://phabricator.wikimedia.org/T157898#3110328 (10Aklapper) Looks like [[ https://github.com/Bitergia/mediawiki-identities/commits/master/wikimedia-affiliations.json | OwlBot also gave up a wee... [17:11:43] (03PS8) 10Joal: Add snapshot to sqoop and namespace_map scripts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/341586 (https://phabricator.wikimedia.org/T160152) [17:13:42] (03PS5) 10Joal: Update static wiki projects list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/342030 (https://phabricator.wikimedia.org/T160153) [17:34:19] (03PS7) 10Joal: Add oozie job for standard metrics computation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/342197 (https://phabricator.wikimedia.org/T160151) [17:42:14] joal: sorry, just saw your ping! anything I can help with? [17:42:45] milimetric: I actually found the thing - Half sqoop, half me :) [17:56:12] (03PS19) 10Joal: Port standard metrics to reconstructed history [analytics/refinery] - 10https://gerrit.wikimedia.org/r/322103 (https://phabricator.wikimedia.org/T160155) (owner: 10Milimetric) [18:50:19] milimetric: still around? [18:59:18] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3110616 (10GLCiampaglia) I am pretty sure my tables can be safely deleted. Thanks for the heads up! Giovanni [19:05:16] joal: sorry, yes [19:05:21] np :) [19:05:23] still randomly not getting pings hm [19:05:27] what's up? [19:05:44] Would you give me 5 minutes brainstorm on oozie? [19:06:01] sure [19:06:13] Thanks :) [20:21:57] 10Analytics, 06Operations, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#3110845 (10Nuria) Design document available in meta: https://meta.wikimedia.org/wiki/Research:PrivacyConsciousABTestingAtWikimediaFoundation [20:22:39] 06Analytics-Kanban, 06Operations, 06Performance-Team, 06Reading-Admin, 10Traffic: Preliminary Design document for A/B testing - https://phabricator.wikimedia.org/T143694#3110846 (10Nuria) Design document available in beta: https://meta.wikimedia.org/wiki/Research:PrivacyConsciousABTestingAtWikimediaFound... [20:23:41] 06Analytics-Kanban, 15User-Elukey: Bump replication factor of system.auth table in cassandra when new nodes have finished bootstrap - https://phabricator.wikimedia.org/T157354#3110851 (10Nuria) 05Open>03Resolved [20:25:51] 06Analytics-Kanban, 15User-Elukey: AQS: Verify that node not being able to restart logs locally to errorlog not to logstash - https://phabricator.wikimedia.org/T155791#3110853 (10Nuria) 05Open>03Resolved [20:26:28] 06Analytics-Kanban, 13Patch-For-Review: Spike: Split unique devices data for Asiacell and non-Asiacell traffic in Iraq - https://phabricator.wikimedia.org/T158237#3110862 (10Nuria) 05Open>03Resolved [20:27:19] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review, 15User-Elukey: Add jmxtrans metrics from Hadoop yarn-mapreduce-historyserver - https://phabricator.wikimedia.org/T156272#3110863 (10Nuria) 05Open>03Resolved [20:27:32] 06Analytics-Kanban, 10Analytics-Wikistats: Simple json model of the site - https://phabricator.wikimedia.org/T157986#3110864 (10Nuria) 05Open>03Resolved [20:27:34] 06Analytics-Kanban, 10Analytics-Wikistats: Visual prototype for community feedback for Wikistats 2.0 iteration 1. - https://phabricator.wikimedia.org/T157827#3110865 (10Nuria) [20:27:47] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations: Reinstall Analytics Hadoop Cluster with Debian Jessie - https://phabricator.wikimedia.org/T157807#3110867 (10Nuria) [20:27:50] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review, 15User-Elukey: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3110866 (10Nuria) 05Open>03Resolved [20:28:31] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 06Services (watching), 15User-mobrovac: EventStreams - https://phabricator.wikimedia.org/T130651#3110871 (10Nuria) [20:28:33] 06Analytics-Kanban, 10EventBus, 10Wikimedia-Stream, 13Patch-For-Review: Create /schema/:schema endpoint in eventbus service to serve schemas by schema_uri - https://phabricator.wikimedia.org/T159179#3110870 (10Nuria) 05Open>03Resolved [20:29:25] 06Analytics-Kanban: Load edit history data into Druid - https://phabricator.wikimedia.org/T131786#3110872 (10Nuria) Seems that we should be able to close this parent task this quarter right? [20:30:00] 06Analytics-Kanban, 10Analytics-Wikistats: Visual Language for http://stats.wikimedia.org replacement - https://phabricator.wikimedia.org/T152033#3110877 (10Nuria) 05Open>03Resolved [20:30:02] 10Analytics, 10Analytics-Wikistats: Design new UI for Wikistats 2.0 - https://phabricator.wikimedia.org/T140000#3110878 (10Nuria) [20:30:33] 06Analytics-Kanban: Add ability to query new AQS endpoint to node pageview API client - https://phabricator.wikimedia.org/T160655#3110880 (10Nuria) a:03fdans [20:37:55] HaeB: did you send a redership metrics as of recent? [20:43:09] nuria: this is the most recent one https://commons.wikimedia.org/wiki/File:Wikimedia_Readership_metrics_for_the_timespan_until_February_5,_2017.pdf [20:43:20] HaeB: k, thank you [21:43:44] 10Analytics, 10Analytics-Dashiki: Change default timeline for browser reports to be recent - https://phabricator.wikimedia.org/T160796#3111134 (10Krinkle) [21:44:07] 10Analytics, 10Analytics-Dashiki: Change default timeline for browser reports to be recent (not 2015) - https://phabricator.wikimedia.org/T160796#3111147 (10Krinkle) [22:00:13] (03CR) 10Milimetric: Add stats for the "New wikitext mode" beta feature (032 comments) [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/343247 (owner: 10Catrope) [22:01:38] (03CR) 10Milimetric: Add stats for the "New wikitext mode" beta feature (031 comment) [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/343247 (owner: 10Catrope) [22:58:24] 10Analytics, 10DBA, 06Operations: Prep to decommission old dbstore hosts (db1046, db1047) - https://phabricator.wikimedia.org/T156844#3111341 (10Tbayer) I have been using `db1047` quite frequently for EventLogging queries as an alternative to `dbstore1002`, either because it was (at times) much faster, or in... [23:08:41] (03PS2) 10Catrope: Add stats for the "New wikitext mode" beta feature [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/343247 [23:09:25] (03CR) 10Milimetric: [V: 032 C: 032] Add stats for the "New wikitext mode" beta feature [analytics/limn-edit-data] - 10https://gerrit.wikimedia.org/r/343247 (owner: 10Catrope)