[02:55:20] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4079623 (10sahil505)
[03:25:48] <wikibugs>	 10Analytics: Stats on which clients/user-agents are making the most API edits - https://phabricator.wikimedia.org/T188549#4079631 (10Whatamidoing-WMF) In terms of those user agents, does the (pasted) list mean that there were hundreds of edits made by bot, a few dozen by the app, but nobody (in that list) using...
[06:34:40] <wikibugs>	 (03CR) 10Amire80: [C: 04-1] Add interlanguage script (031 comment) [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (owner: 10Milimetric)
[07:13:07] <wikibugs>	 (03PS2) 10Amire80: Add interlanguage script [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[07:13:39] <wikibugs>	 (03CR) 10Amire80: [C: 04-1] Add interlanguage script (031 comment) [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[07:56:46] <wikibugs>	 (03CR) 10Amire80: [C: 04-1] "The pageviews part was getting all the pageviews, including spiders and mobile views. I only need human desktop pageviews. Because of that" [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[11:06:56] * elukey lunch!
[12:38:55] <wikibugs>	 10Analytics, 10Analytics-Cluster: Requesting account expiration extension - https://phabricator.wikimedia.org/T183291#4080361 (10elukey) a:05elukey>03None
[12:39:14] <wikibugs>	 10Analytics, 10Analytics-Cluster: Requesting account expiration extension - https://phabricator.wikimedia.org/T183291#3849363 (10elukey) Since https://phabricator.wikimedia.org/T184085 is resolved, should we keep this task open or can we close?
[12:43:53] <fdans>	 elukey: whenever you have a moment let's nullify the QuickSurvey data in master?
[12:45:21] <elukey>	 fdans: sure! lemme check with Manuel first
[12:45:31] <fdans>	 thank youuu
[12:46:18] <elukey>	 ah no he is on holiday
[12:48:15] <fdans>	 so we wait until he's back elukey ?
[12:49:04] <elukey>	 fdans: I am alerting the db chan, I think we can procedeed but I don't find the task :)
[12:49:08] <elukey>	 do you have it handy?
[12:49:28] <elukey>	 https://phabricator.wikimedia.org/T174386
[12:49:30] <elukey>	 found it
[12:49:30] <fdans>	 https://phabricator.wikimedia.org/T174386
[12:49:33] <fdans>	 ha, yeah
[12:52:16] <elukey>	 fdans: batcave?
[12:52:26] <fdans>	 yessir
[13:07:20] <icinga-wm>	 PROBLEM - Check status of defined EventLogging jobs on eventlog1002 is CRITICAL: CRITICAL: Stopped EventLogging jobs: eventlogging-consumer@mysql-m4-master-00 eventlogging-consumer@mysql-eventbus
[13:09:19] <fdans>	 !log stopped 2 mysql consumers as precaution for T174386
[13:09:21] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:09:21] <stashbot>	 T174386: Remove sensitive fields from whitelist for QuickSurvey schemas (end of Q2) - https://phabricator.wikimedia.org/T174386
[13:09:33] <ottomata>	 o/
[13:09:58] <elukey>	 ottomata: sorry for the page, my fault :)
[13:10:02] <elukey>	 didn't silence el1002
[13:10:09] <ottomata>	 np!
[13:21:53] <fdans>	 elukey joal shall we merge this? https://gerrit.wikimedia.org/r/#/c/420346/
[13:22:05] <joal>	 Pease go ahead fdans :)
[13:22:17] <wikibugs>	 (03CR) 10Fdans: [V: 032 C: 032] Add hi.wikimedia, zh.wikidata and commons.wikipedia to whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/420346 (owner: 10Fdans)
[13:23:45] <elukey>	 sure :)
[13:23:52] <elukey>	 Burrow is me and Fran people
[13:28:30] <ottomata>	 !log restarting kafka mirror maker main -> jumbo using new consumer
[13:28:32] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:34:53] <ottomata>	 ahhh shoot elukey, new consumer metrics change jmx beans
[13:35:02] <ottomata>	 should be the same for new kafka versions, sohm
[13:35:08] <ottomata>	 maybe i could switch it to prometheus now?
[13:39:42] <elukey>	 ottomata: it could be an option yes!
[13:41:28] <elukey>	 ottomata: whenever you have time we should also check https://phabricator.wikimedia.org/T190400
[13:41:37] <ottomata>	 oh yes
[13:41:41] <ottomata>	 i half looked at those
[13:41:42] <ottomata>	 will do now
[13:52:31] <wikibugs>	 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#4080629 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by oblivian on neodymium.eqiad.wmnet for hosts: ``` ['conf1005.eqiad.wmnet'...
[13:56:15] <ottomata>	 oh elukey i have already readied mm for prometheus, woo hoo!  I just never used it because we haven't used new mirror maker profile yet
[13:56:18] <ottomata>	 because it was incompatilble
[13:56:21] <ottomata>	 i'll just make it compatible... :)
[13:56:44] <elukey>	 ottomata: hahaha yes I thought you remembered! 
[13:56:58] <elukey>	 past you made something great and you didn't remember :)
[13:57:02] <ottomata>	 indeed!
[13:57:03] <ottomata>	 thanks past me!
[14:12:53] <moritzm>	 I don't have time to review https://gerrit.wikimedia.org/r/#/c/421891 today, but will do so tomorrow
[14:17:30] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Add hi.wikimedia, zh.wikidata and commons.wikipedia to whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/420346 (owner: 10Fdans)
[14:18:46] <fdans>	 milimetric: helloooo do you have comments on the carousel/add more rows change?
[14:18:57] <milimetric>	 fdans: gonna review this morning
[14:19:06] <milimetric>	 well, maybe after meetings, is that ok?
[14:20:22] <ottomata>	 np thanks moritzm
[14:23:26] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4080691 (10fdans)
[14:23:28] <wikibugs>	 10Analytics-Kanban: The size of metric areas in the dashboard should scale to available window space - https://phabricator.wikimedia.org/T187345#4080693 (10fdans)
[14:25:33] <wikibugs>	 (03PS3) 10Amire80: WIP Analyzing failed ULS searches [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/413270 (https://phabricator.wikimedia.org/T190630)
[14:25:34] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#3956038 (10fdans)
[14:25:36] <wikibugs>	 10Analytics-Kanban: Show compact view of metrics in detail page when visited from mobile - https://phabricator.wikimedia.org/T188013#4080701 (10fdans)
[14:26:49] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4080721 (10fdans)
[14:26:52] <wikibugs>	 10Analytics-Kanban: Create metrics dropdown component - https://phabricator.wikimedia.org/T188526#4080723 (10fdans)
[14:26:57] <ottomata>	 !log restarting jumbo -> eqiad mirror makers with prometheus instead of jmx
[14:26:59] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:28:21] <wikibugs>	 10Analytics-Kanban: Adapt TableChart to compact size - https://phabricator.wikimedia.org/T188952#4080737 (10fdans)
[14:28:23] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#3956038 (10fdans)
[14:29:44] <wikibugs>	 10Analytics-Kanban: Margins change when going back from detail page to dashboard (mobile) - https://phabricator.wikimedia.org/T188747#4080747 (10fdans)
[14:29:46] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4080745 (10fdans)
[14:30:15] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#3956038 (10fdans) Merged in all tasks that were resolved by the changes in Changeset https://gerrit.wikimedia.org/r/#/c/416999/
[14:31:18] <ottomata>	 elukey: , got a sec to help me with prometheus dashboarding?
[14:31:50] <elukey>	 ottomata: sure!
[14:31:55] <wikibugs>	 (03CR) 10Milimetric: "Amire80, do take over the patch, I'll merge it when it's doing what you need.  I was waiting for you to verify and add more scripts." (031 comment) [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[14:31:57] <ottomata>	 bc?
[14:32:06] <elukey>	 sure
[14:32:19] <elukey>	 1 min that I was brewing coffee
[14:33:04] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4080765 (10fdans)
[14:33:10] <wikibugs>	 (03CR) 10Mforns: "Hey Amit!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421708 (https://phabricator.wikimedia.org/T190435) (owner: 10Amitjoki)
[14:34:17] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#3956038 (10fdans) a:03fdans
[14:36:07] <wikibugs>	 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey, 10User-Joe: rack/setup/install conf1004-conf1006 - https://phabricator.wikimedia.org/T166081#4080780 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by oblivian on neodymium.eqiad.wmnet for hosts: ``` ['conf1006.eqiad.wmnet'...
[14:53:17] <wikibugs>	 (03PS3) 10Amire80: Add interlanguage script [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[14:53:31] <wikibugs>	 (03CR) 10Amire80: [C: 031] Add interlanguage script [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[15:00:47] <wikibugs>	 10Quarry, 10cloud-services-team, 10Patch-For-Review: Quarry server errors caused by Cloud VPS shared proxy failures - https://phabricator.wikimedia.org/T190218#4080890 (10Bstorm) So, I'm still manually truncating this log because our logrotate cron runs from cron.daily :-p I plan to puppetize a move of that...
[15:03:20] <joal>	 milimetric: standup?
[15:03:21] <ottomata>	 milimetric:  standup?
[15:06:46] <wikibugs>	 (03CR) 10Mforns: [C: 04-1] "Hey Amit! The code works, but it could be simpler (and better) :] See comments inline!" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421711 (owner: 10Amitjoki)
[15:10:56] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on eventlog1002 is OK: OK: All defined EventLogging jobs are runnning.
[15:12:21] <elukey>	 !log restart eventlogging mysql consumers after maintenance
[15:12:23] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:15:06] <wikibugs>	 (03CR) 10Milimetric: [C: 032] Add interlanguage script [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[15:15:13] <wikibugs>	 (03Merged) 10jenkins-bot: Add interlanguage script [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/416976 (https://phabricator.wikimedia.org/T158835) (owner: 10Milimetric)
[15:41:15] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Remove sensitive fields from whitelist for QuickSurvey schemas (end of Q2) - https://phabricator.wikimedia.org/T174386#4080947 (10elukey) Update queries executed on db1108 and db1107, all good!
[15:56:11] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4080988 (10fdans)
[15:57:19] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4077070 (10fdans) p:05Triage>03High
[15:58:50] <wikibugs>	 10Analytics, 10Analytics-Kanban: Correct documentation for the referer_class field in pageview_hourly - https://phabricator.wikimedia.org/T190579#4080996 (10fdans) p:05Triage>03Unbreak! a:03JAllemandou
[15:59:49] <wikibugs>	 10Analytics, 10Easy: Productionize job for Global Innovation Index from Hadoop Geowiki data - https://phabricator.wikimedia.org/T190535#4081005 (10fdans) p:05Normal>03Low
[16:01:05] <wikibugs>	 10Analytics: Phasing away one of the mobile apps session metrics jobs. - https://phabricator.wikimedia.org/T190459#4081012 (10fdans) p:05Triage>03High
[16:01:22] <wikibugs>	 10Analytics, 10Analytics-Kanban: Phasing away one of the mobile apps session metrics jobs. - https://phabricator.wikimedia.org/T190459#4074265 (10fdans) a:03JAllemandou
[16:02:58] <wikibugs>	 10Analytics: Support R Kernels by default for all users. - https://phabricator.wikimedia.org/T190453#4081022 (10fdans) p:05Triage>03Low
[16:03:23] <wikibugs>	 10Analytics: Spark notebook integration - https://phabricator.wikimedia.org/T190443#4081025 (10fdans) p:05Triage>03Low
[16:05:18] <wikibugs>	 10Analytics, 10Analytics-Kanban: Issues with page deleted dates on data lake - https://phabricator.wikimedia.org/T190434#4073560 (10fdans) p:05Triage>03Normal
[16:11:33] <wikibugs>	 10Analytics, 10Analytics-Kanban: Issues with page deleted dates on data lake - https://phabricator.wikimedia.org/T190434#4073560 (10Milimetric) And yes, the end_timestamp and revision_deleted_timestamp should match and match the point in time the page was deleted, as recorded in the logging table.
[16:12:14] <wikibugs>	 10Analytics: Update wikimedia-history revision data with deleted field (and find it a new name?) - https://phabricator.wikimedia.org/T178587#4081097 (10JAllemandou) p:05High>03Triage
[16:13:59] <wikibugs>	 10Analytics, 10MobileFrontend, 10Performance-Team, 10Readers-Web-Backlog, 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381#4081113 (10fdans) We think @Krinkle's question is better addressed by @Tgr
[16:17:20] <wikibugs>	 10Analytics: Update wikimedia-history revision data with deleted field (and find it a new name?) - https://phabricator.wikimedia.org/T178587#4081128 (10fdans) p:05Triage>03High
[16:18:47] <wikibugs>	 10Analytics, 10Performance-Team: Archive Kasocki repository - https://phabricator.wikimedia.org/T190365#4071243 (10fdans) ping @Aklapper do you have permissions to do these?
[16:20:06] <wikibugs>	 10Analytics: Some metrics don't work in the topic selector - https://phabricator.wikimedia.org/T188268#4081141 (10fdans) p:05Triage>03High
[16:22:33] <wikibugs>	 10Analytics: Map tooltip and line graph guide are misaligned in Ubuntu Chrome - https://phabricator.wikimedia.org/T187453#4081149 (10fdans) p:05Triage>03Normal
[16:23:29] <wikibugs>	 10Analytics: We should prevent the user from trying to rediscover America - https://phabricator.wikimedia.org/T187452#4081153 (10fdans) p:05Triage>03Normal
[16:26:31] <wikibugs>	 10Analytics: Investigate AQS cassandra schema hash warninga - https://phabricator.wikimedia.org/T178832#4081156 (10fdans) p:05Triage>03Low
[16:27:38] <wikibugs>	 (03PS1) 10Joal: Correct pageview_hourly referer_class comment [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421940 (https://phabricator.wikimedia.org/T190579)
[16:29:46] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Correct documentation for the referer_class field in pageview_hourly - https://phabricator.wikimedia.org/T190579#4081176 (10JAllemandou)
[16:31:15] <wikibugs>	 10Analytics, 10Analytics-Kanban: Gather all constants related to mobile/responsiveness in config - https://phabricator.wikimedia.org/T190339#4081185 (10fdans) p:05Triage>03High a:03fdans
[16:32:41] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace all hover actions with touch-compatible UI - https://phabricator.wikimedia.org/T188277#4081199 (10fdans) a:03fdans
[16:32:53] <wikibugs>	 10Analytics, 10Analytics-Kanban: Phasing away one of the mobile apps session metrics jobs. - https://phabricator.wikimedia.org/T190459#4074265 (10JAllemandou)
[16:33:15] <wikibugs>	 10Analytics, 10Analytics-Kanban: Replace all hover actions with touch-compatible UI - https://phabricator.wikimedia.org/T188277#4002130 (10fdans) p:05Triage>03High
[16:39:39] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Automate creation of sqoop list of wikis to import data for - https://phabricator.wikimedia.org/T190700#4081260 (10fdans) p:05Triage>03High
[16:39:50] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Automate creation of sqoop list of wikis to import data for from sitematrix - https://phabricator.wikimedia.org/T190700#4081260 (10fdans)
[16:40:20] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats 2.0: "aa.wikipedia.org" exists and has data available, but marked "Invalid" - https://phabricator.wikimedia.org/T187414#4081272 (10fdans) p:05Triage>03Normal
[16:42:17] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0: Page heading style varies - https://phabricator.wikimedia.org/T187412#4081278 (10fdans) p:05Triage>03Normal a:03fdans
[16:45:11] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Wikistats 2: New Pages split by editor type wrongly claims no anonymous users create pages - https://phabricator.wikimedia.org/T185342#3913500 (10fdans) p:05Low>03Normal
[16:47:01] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Tracking: Update client-side event validator to support (at least) draft 3 of JSON Schema - https://phabricator.wikimedia.org/T182094#3812560 (10fdans) p:05Triage>03Low
[16:51:03] <wikibugs>	 10Analytics, 10Wikimedia-Stream, 10Wikimedia-Incident: Alerts for common/important EventStreams topic volume - https://phabricator.wikimedia.org/T174493#4081322 (10fdans) p:05High>03Low
[16:51:05] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Tracking: Update client-side event validator to support (at least) draft 3 of JSON Schema - https://phabricator.wikimedia.org/T182094#4081325 (10Milimetric)
[16:51:09] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Tracking: Use draft 4 of JSON Schema specification - https://phabricator.wikimedia.org/T46809#4081328 (10Milimetric)
[16:52:04] <wikibugs>	 (03PS1) 10Joal: Update mobile-apps session metrics oozie [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421946 (https://phabricator.wikimedia.org/T190459)
[16:53:56] <wikibugs>	 10Analytics: Order hardware labs storage for mediawiki history analytics friendly DB - https://phabricator.wikimedia.org/T175604#4081346 (10fdans) p:05Triage>03Low
[16:54:37] <wikibugs>	 10Analytics: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059#4061513 (10fdans) p:05Triage>03Normal
[16:56:16] <wikibugs>	 10Analytics: Update Mediawiki Table manuals on wiki - https://phabricator.wikimedia.org/T179407#3723665 (10fdans) p:05Triage>03Normal
[16:57:11] <wikibugs>	 10Analytics: Make 'metric' field not a partition in mediawiki_metrics - https://phabricator.wikimedia.org/T190058#4081380 (10fdans) p:05Triage>03High
[16:58:11] <wikibugs>	 (03PS2) 10Joal: Update mobile-apps session metrics oozie [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421946 (https://phabricator.wikimedia.org/T190459)
[16:59:59] <wikibugs>	 10Analytics: Create reports in wikistats for "most prolific editors" - https://phabricator.wikimedia.org/T189882#4081390 (10fdans) p:05Triage>03Normal
[17:02:11] <wikibugs>	 10Analytics, 10Analytics-General-or-Unknown, 10Language-Team, 10Mobile-Apps, and 3 others: there should be a comparison of clicks count on interlanguage links on different platforms - https://phabricator.wikimedia.org/T78351#843122 (10Milimetric) @Amire80 is this done after we publish the dashboard we're w...
[17:28:37] * elukey off!! byyee
[17:36:49] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4081576 (10Nuria) >We can increase the dispatch interval from 60 seconds to a higher number (e.g. 120 seconds) so that the tracker would dispatch events less...
[17:38:49] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v5.8.0-Manatee-On-A-Skateboard: Decrease the request from iOS app to bohrium - https://phabricator.wikimedia.org/T190566#4081582 (10JMinor)
[17:38:50] <wikibugs>	 (03CR) 10Nuria: "You mean hard to "make" monthly queries on top of daily data?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421616 (https://phabricator.wikimedia.org/T190409) (owner: 10Milimetric)
[17:45:01] <nuria_>	 milimetric: i think we should label geowiki data so it is "easily" findable on pivot:
[17:45:37] <nuria_>	 milimetric:  does that seem ok?
[17:45:48] <milimetric>	 nuria_: I'm changing the way it's loaded, but of course, I'll add metadata once it's stable
[17:47:02] <milimetric>	 nuria_: in your response to your comment on the task, yeah, I talked about it with joseph, and this makes sense, leaving the daily data in Hadoop only and loading easier-to-digest stuff in druid, makes the superset stuff less nasty and makes sanitizing easier going forward
[17:47:09] <milimetric>	 but let me know if you want to re-visit
[17:47:42] <nuria_>	 milimetric: sounds good, what is is what you are changing? keeping daily data on hive and loading monthly data into pivot?
[17:47:51] <nuria_>	 milimetric: sounds good, what is is what you are changing? keeping daily data on hive and loading monthly data into DRUID?
[17:47:55] <nuria_>	 sorry, druid
[17:48:37] <nuria_>	 cc joal 
[17:48:43] <milimetric>	 yes nuria_, exactly, so one more workflow step in the oozie job
[17:48:57] <milimetric>	 gonna send the change today after I cr fran’s
[17:49:05] <nuria_>	 milimetric: then there is no need to label it in pivot as it cannot deal with monthly resolution
[17:49:22] <joal>	 very true nuria_ 
[17:49:48] <nuria_>	 milimetric: so nevermind, we will be able to look at it just with superset (which is fine, same than unique devices)
[17:51:11] <milimetric>	 well, I’m going to delete the daily one from druid, load the monthly one into druid, and label the monthly one accordingly
[17:51:52] <wikibugs>	 (03CR) 10Nuria: "Looks good! let's do a test run to make sure we did not missed any typos." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421946 (https://phabricator.wikimedia.org/T190459) (owner: 10Joal)
[17:52:30] <nuria_>	 milimetric: what i am saying is that the label is not needed as that data in pivot will not be "nicely viewable" due to monthly resolution 
[17:53:52] <wikibugs>	 (03CR) 10Nuria: [C: 031] "Let's also update : https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Pageview_hourly" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421940 (https://phabricator.wikimedia.org/T190579) (owner: 10Joal)
[17:54:29] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Correct documentation for the referer_class field in pageview_hourly - https://phabricator.wikimedia.org/T190579#4077428 (10Nuria) Note that this change will update table docs on code but not in hive meta store, for latest docs : https://wikitech.wikimedia...
[17:56:57] <milimetric>	 oh, that needs the pivot upgrade, riiiight, good point nuria
[17:59:56] <wikibugs>	 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4081683 (10Tbayer)
[18:05:01] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4081723 (10Nuria) Snipets:  <!-- Piwik --> <script type="text/javascript">   var _paq = _paq || [];   _paq.push(["setDomains", ["*.design.wikimedia.org."]]);   _paq.pus...
[18:06:22] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4081738 (10Nuria) I gave access to reserach user (since i think you mentioned you already had that one)
[18:06:39] <wikibugs>	 10Analytics-Kanban, 10WikimediaUI Style Guide: Setup & integrate analytics on design.wikimedia.org - https://phabricator.wikimedia.org/T188786#4081739 (10Nuria)
[18:07:56] <wikibugs>	 (03CR) 10Joal: [V: 032 C: 032] "Self merging after having updated docs and prod table" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421940 (https://phabricator.wikimedia.org/T190579) (owner: 10Joal)
[18:08:19] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Correct documentation for the referer_class field in pageview_hourly - https://phabricator.wikimedia.org/T190579#4081744 (10JAllemandou)
[18:14:42] <wikibugs>	 (03CR) 10Joal: "Tested" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421946 (https://phabricator.wikimedia.org/T190459) (owner: 10Joal)
[18:15:19] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Phasing away one of the mobile apps session metrics jobs. - https://phabricator.wikimedia.org/T190459#4081751 (10JAllemandou)
[18:16:09] <wikibugs>	 (03CR) 10Nuria: [V: 032 C: 032] Update mobile-apps session metrics oozie [analytics/refinery] - 10https://gerrit.wikimedia.org/r/421946 (https://phabricator.wikimedia.org/T190459) (owner: 10Joal)
[18:16:49] <joal>	 Hey ottomata - Can I ask for a favor?
[18:19:02] <ottomata>	 ya
[18:19:07] <ottomata>	 hey joal wassup?
[18:24:36] <joal>	 Hay ottomata 
[18:24:55] <joal>	 ottomata: Could we try to install python-protobuf onto notebook1003?
[18:26:35] <ottomata>	 ya sure
[18:26:45] <joal>	 \o/ ottomata :)
[18:26:46] <joal>	 Thanks
[18:26:47] <ottomata>	 joal:  just curious, what for?
[18:27:03] <joal>	 ottomata: I want to try facets
[18:27:08] <joal>	 https://github.com/PAIR-code/facets
[18:28:01] <ottomata>	 joal:  installed, puppetizing now
[18:35:12] <joal>	 testing ottomata 
[18:35:45] <joal>	 By the way ottomata - Im playing with lab now - Pretty awesome !!
[18:37:15] <ottomata>	 yeahhh!
[18:37:16] <ottomata>	 cool
[18:42:56] <nuria_>	 ottomata: url errors are logged from client: ./modules/ext.eventLogging.core.js:				mw.eventLog.logFailure( schemaName, 'urlSize' );
[18:43:33] <nuria_>	 ottomata: so there is one  error every 4 secs
[18:43:54] <nuria_>	 ottomata: at 150 per sec is not very much 
[18:45:16] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 3 others: Expose a metric that reflect EventBus queue pressure - https://phabricator.wikimedia.org/T190416#4072972 (10Pchelolo) We can fetch the data from burrow as well, but the current version of burrow installed in our production is not s...
[18:45:18] <joal>	 ottomata: I fail to use facets :(
[18:45:28] <joal>	 ottomata: is there a way to get access to logs of nebooks?>
[18:48:40] <nuria_>	 other question for ottomata : is there a jumbo kafka dashboard?
[18:53:49] <ottomata>	 nuria_:  ya
[18:53:50] <ottomata>	 https://grafana-admin.wikimedia.org/dashboard/db/prometheus-kafka?refresh=5m&orgId=1
[18:53:55] <ottomata>	 also
[18:53:55] <ottomata>	 https://grafana-admin.wikimedia.org/dashboard/db/kafka-by-topic-prometheus?from=now-3h&to=now&refresh=5m&orgId=1
[18:54:10] <ottomata>	 joal:  hm
[18:54:19] <ottomata>	 yes, but i would think all the logs are on your side
[18:54:26] <ottomata>	 what are you trying to do?
[18:54:28] <joal>	 mwarf
[18:59:11] <joal>	 ottomata: I'm opening the facets example, and i doesn't error but doesn't show anything :(
[18:59:14] <joal>	 sadness
[19:00:05] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 3 others: Expose a metric that reflect EventBus queue pressure - https://phabricator.wikimedia.org/T190416#4081887 (10Pchelolo) With the new queue, the number of pending jobs in the queue will not be an issue, Kafka doesn't really care how m...
[19:00:08] <joal>	 ottomata: I managed to have working at home by installing the extension
[19:02:39] <ottomata>	 joal: 
[19:02:46] <ottomata>	 i've been able to install the nbextension
[19:02:47] <ottomata>	 what next?
[19:03:27] <joal>	 try to open and run the example notebook in facets_overview
[19:03:34] <joal>	 Within chrome
[19:05:32] <ottomata>	 hmm joal, nothing happens, right?
[19:05:37] <joal>	 Correct, same for me ottomata 
[19:05:47] <joal>	 It should show you a beautifull app
[19:06:20] <joal>	 IIRC ottomata I had to restart jupyter after having installed the extension - But not sure
[19:06:42] <joal>	 ottomata: Completely different matter - There is something I don't understand in spark ...
[19:06:59] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Change '--' to something more helpful in Wikistats page views by country table view - https://phabricator.wikimedia.org/T187427#4081911 (10mforns) a:03sahil505
[19:10:08] <ottomata>	 joal:  dunno what's going on really, but i think it is something with the load of facets-dist/facets-jupyter.html
[19:10:11] <ottomata>	 not sure if that is right
[19:10:17] <ottomata>	 protostr has a value
[19:10:26] <joal>	 hm
[19:10:38] <ottomata>	 joal:  what's up with spark?
[19:11:55] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 3 others: Expose a metric that reflect EventBus queue pressure - https://phabricator.wikimedia.org/T190416#4072972 (10EBernhardson) >>! In T190416#4081887, @Pchelolo wrote: > With the new queue, the number of pending jobs in the queue will n...
[19:12:15] <joal>	 I experience skewed computation, but without explicit skewed data :S
[19:12:20] <joal>	 ottomata: --^
[19:12:42] <ebernhardson>	 ouch :(
[19:13:07] <joal>	 ottomata: For a certain job, every tasks take ~4 minutes, and 2 of them takes 1h20 ...
[19:13:18] <ottomata>	 hm
[19:13:20] <joal>	 But the datasize they work is equivalent, as well as the number of items
[19:13:21] <ottomata>	 any pattern?
[19:13:26] <ottomata>	 like a single host doing that?
[19:13:29] <joal>	 nope
[19:13:34] <joal>	 2 different hosts
[19:14:56] <joal>	 It feel super weird
[19:15:52] <joal>	 ottomata: I'm wondering if I should trust the spark-UI
[19:17:29] <ebernhardson>	 joal: fwiw, i've seen similar with window functions that performed bad on mildly-skewed data. I had sum(x) over (...) and with most partitions being ~100 rows and some being ~50k rows (in a dataset of 100M+). The 50k sessions wern't big enough to make overall partitions noticably different sizes
[19:18:42] <joal>	 ebernhardson: If you have a minute to brainstorm ?
[19:18:47] <joal>	 ebernhardson: https://yarn.wikimedia.org/proxy/application_1521803817821_9123/stages/
[19:19:04] <joal>	 In completed stages, one stage takes 1.2h (others merely minues
[19:20:25] <joal>	 In that huge stage that takes 1.2h, you can look at detailed tasks: every take max 3min, except 2 that take 1.2h
[19:20:44] <joal>	 And thei number of worked items or data read/split is equivalent
[19:20:52] * joal doesn't understand
[19:20:58] <ebernhardson>	 joal: sure, looking
[19:27:47] <wikibugs>	 10Analytics-Kanban, 10Operations, 10Patch-For-Review: Review changes to /etc/java-8-openjdk/security/java.security in Kafka from u162 update - https://phabricator.wikimedia.org/T190400#4082018 (10Ottomata) p:05Triage>03Normal a:03Ottomata
[19:28:06] <ebernhardson>	 joal: sadly, i dunno :( It looks like you have a leftOuterJoin in the partitioner which is known to do badly on skewed data, i'd look at the distributions perhaps of the two sides of that join on the key?
[19:28:34] <joal>	 ebernhardson: makes sense - Still super weird
[19:28:53] <ebernhardson>	 although actually, i see spark has a ticket 'resolved' for 2.1 about skewed join performance
[19:48:48] <nuria_>	 bearloga: +1 to that data science JD description
[19:49:53] <bearloga>	 nuria_: :D
[19:54:54] <wikibugs>	 (03CR) 10Mforns: [V: 032 C: 032] "Looks great :]" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421330 (https://phabricator.wikimedia.org/T187427) (owner: 10Sahil505)
[20:05:29] <milimetric>	 ottomata: can you merge https://gerrit.wikimedia.org/r/#/c/421920/
[20:05:58] <wikibugs>	 10Analytics, 10MobileFrontend, 10Readers-Web-Backlog, 10Performance-Team (Radar), 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381#4082177 (10Imarlier)
[20:07:31] <wikibugs>	 10Analytics, 10Performance-Team (Radar): Archive Kasocki repository - https://phabricator.wikimedia.org/T190365#4082182 (10Krinkle)
[20:07:52] <wikibugs>	 10Analytics, 10Performance-Team (Radar): Archive Kasocki repository - https://phabricator.wikimedia.org/T190365#4071243 (10Krinkle) p:05Triage>03Low
[20:08:05] <wikibugs>	 10Analytics, 10MobileFrontend, 10Readers-Web-Backlog, 10Performance-Team (Radar), 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381#4082192 (10Krinkle) p:05Triage>03Normal
[20:18:38] <ottomata>	 milimetric:  done
[20:18:43] <milimetric>	 thanks ottomata 
[20:21:33] <joal>	 ebernhardson: There must have been something weird happening with the partitioner - I changed the number of partition (Made it 4 times bigger) - Skew gone! 
[20:21:40] <joal>	 grumble grumble
[20:27:18] <wikibugs>	 10Analytics, 10CirrusSearch, 10Discovery, 10Discovery-Search, and 3 others: Expose a metric that reflect EventBus queue pressure - https://phabricator.wikimedia.org/T190416#4082264 (10dcausse) Here is the original need: - have a continuous process that checks the elastic index consistency for every single...
[20:32:07] <wikibugs>	 10Analytics-Tech-community-metrics, 10Gerrit, 10Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#4082284 (10Paladox) 05Open>03Resolved https://gerrit.wikimedia.org/r/#/c/99101/ is now working :)
[20:55:05] <wikibugs>	 (03PS1) 10Milimetric: Make geowiki job output monthly druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422060 (https://phabricator.wikimedia.org/T190409)
[20:55:51] <wikibugs>	 (03CR) 10Milimetric: "This is ready for review, but I do have to test it to double check it's functional, so I'll update with any problems from that." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422060 (https://phabricator.wikimedia.org/T190409) (owner: 10Milimetric)
[21:04:09] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Checklist for geowiki pipeline - https://phabricator.wikimedia.org/T190409#4082381 (10Milimetric)
[21:06:13] <wikibugs>	 10Analytics, 10Analytics-Wikistats: Present a page view metric description to the user that they are likely to understand - https://phabricator.wikimedia.org/T182109#4082385 (10mforns) @Tbayer  > Given that, I would strongly advise against creating yet another new page in yet another location. I suggest to ins...
[21:06:42] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo - https://phabricator.wikimedia.org/T189464#4082386 (10Ottomata) --new.consumer seems to behave wayyyy better than the old one.  No wonder t...
[21:15:17] <wikibugs>	 (03CR) 10Nuria: [C: 031] Make geowiki job output monthly druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/422060 (https://phabricator.wikimedia.org/T190409) (owner: 10Milimetric)
[21:16:00] <nuria_>	 chelsyx: please take a look at this data https://gerrit.wikimedia.org/r/#/c/422060/ it is similar that what we were talking about the other day that you were interested in putting into druid
[21:17:52] <chelsyx>	 nuria_: Thanks!
[21:23:49] <nuria_>	 chelsyx: it will not go back in time as to the beginning on mediawiki  history but i think it will be useful as of now, it shall be available via superset dashboard
[21:27:09] <wikibugs>	 10Analytics-Kanban, 10Google-Summer-of-Code (2018): [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189210#4082438 (10Nuria)
[21:27:11] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4082439 (10Nuria)
[21:27:14] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: The alert message about adblocker is not fully shown on smaller screens - https://phabricator.wikimedia.org/T188208#4082437 (10Nuria) 05Open>03Resolved
[21:27:26] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Alarm when /mnt/hdfs is mounted but showing no files/dirs in there - https://phabricator.wikimedia.org/T187073#4082440 (10Nuria) 05Open>03Resolved
[21:27:50] <wikibugs>	 10Analytics-Kanban, 10User-Elukey: Reduction of stat1005's disk space usage - https://phabricator.wikimedia.org/T186776#4082442 (10Nuria) 05Open>03Resolved
[21:28:09] <wikibugs>	 10Analytics-Kanban: Operational improvements and maintenance in EventLogging in Q4 {oryx} - https://phabricator.wikimedia.org/T130247#4082444 (10Nuria)
[21:28:11] <bearloga>	 ottomata: do you think it'd be possible to get RStudio Server (https://www.rstudio.com/products/RStudio/#Server, see https://rstudio.cloud/ for example of it in production) running on the notebook machines while you're still working on those or are you just doing the hardware refresh + jupyterhub stuff and this would be way out of scope?
[21:28:14] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade eventlogging servers to Stretch - https://phabricator.wikimedia.org/T114199#4082443 (10Nuria) 05Open>03Resolved
[21:28:30] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade eventlogging servers to Stretch - https://phabricator.wikimedia.org/T114199#1687834 (10Nuria)
[21:29:11] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats, 10Hindi-Sites, 10Patch-For-Review: Hindi Wikiversity is not showing in Wikimedia Stats - https://phabricator.wikimedia.org/T183682#4082449 (10Nuria) 05Open>03Resolved
[21:29:35] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade eventlogging servers to Stretch - https://phabricator.wikimedia.org/T114199#1687834 (10Nuria)
[21:29:38] <wikibugs>	 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), and 2 others: Deprecation of mw.errors.* metrics - https://phabricator.wikimedia.org/T188749#4082451 (10Nuria) 05Open>03Resolved
[21:29:42] <wikibugs>	 10Analytics-Kanban, 10ChangeProp, 10EventBus, 10Patch-For-Review, and 2 others: Export burrow metrics to prometheus - https://phabricator.wikimedia.org/T180442#4082453 (10Nuria) 05Open>03Resolved
[21:30:04] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Get fancy with type casting in Refine job - https://phabricator.wikimedia.org/T189332#4082454 (10Nuria) 05Open>03Resolved
[21:31:24] <wikibugs>	 10Analytics-Kanban: Allow switching metrics in a dashboard widget - Carrousel - https://phabricator.wikimedia.org/T187440#4082456 (10Nuria)
[21:39:45] <wikibugs>	 (03CR) 10Nuria: "This patch still does not render -at all-  for me" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[21:40:02] <nuria_>	 milimetric: have you tried fdans patch? 
[21:40:15] <nuria_>	 milimetric: this one does not render for me: https://gerrit.wikimedia.org/r/#/c/421001/
[21:41:25] <wikibugs>	 (03CR) 10Nuria: "Plenty of console warnings: "property or method "getCarouselOffset" is not defined on the instance but referenced during render. Make sure" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[21:47:29] <wikibugs>	 (03CR) 10Nuria: [C: 04-1] Metrics carousel (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/421001 (https://phabricator.wikimedia.org/T187440) (owner: 10Fdans)
[21:49:12] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: [EL] Correct Print schema mysql whitelist - https://phabricator.wikimedia.org/T190223#4082555 (10Nuria) 05Open>03Resolved
[21:49:29] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Port Kafka clients to new jumbo cluster - https://phabricator.wikimedia.org/T175461#4082557 (10Nuria)
[21:49:32] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. - https://phabricator.wikimedia.org/T185136#4082556 (10Nuria) 05Open>03Resolved
[22:15:46] <wikibugs>	 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4082747 (10Nuria)
[22:15:48] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Include X-Client-IP in EventLogging data and geocode during Hive JSON Refinement - https://phabricator.wikimedia.org/T186833#4082746 (10Nuria) 05Open>03Resolved
[22:29:36] <nuria_>	 bearloga: you can add that request to the backlog for next year
[22:30:11] <nuria_>	 bearloga: as a subtask https://phabricator.wikimedia.org/T188275
[22:33:27] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4082835 (10Nuria) @fdans: I think there are issues with the code that was merged to master . This is what i see now on my desktop:  {F16153296}  cc @Milimetric , i think is probably wise to...
[22:44:02] <wikibugs>	 10Analytics-Kanban, 10Analytics-Wikistats: Make the Wikistats 2 UI responsive - https://phabricator.wikimedia.org/T186812#4082883 (10Nuria) I think it is my zoom level in chrome what makes see it this way, also think that the thresholds for 3 versus two slots might need a bit of work
[23:47:43] <bearloga>	 nuria_ ottomata (and mforns): don't look into stuff with T172581#4082983 yet. I pinged dr0ptp4kt about it and we'll take a look ourselves first but heads up that I may need your help if we can't fix it.
[23:47:43] <stashbot>	 T172581: Set up mechanism for archiving Google Search Console data - https://phabricator.wikimedia.org/T172581