[07:36:00] morninggggg [07:36:04] joal: o/ [07:37:01] whenever you want we can test the --use-segment-metadata [08:01:03] Hi elukey :) [08:11:18] joal: pivot restarted with the new option [08:11:26] Thanks mate :) [08:11:30] checking [08:14:25] not working elukey :( [08:14:27] Mwarf [08:17:20] elukey: Looks like I'm gonna have to write Pivot config myself :S [08:18:19] joal: I tried /usr/bin/nodejs /srv/deployment/analytics/pivot/deploy/src/bin/pivot --help but can't see the option in there.. [08:20:21] Can't find any info elukey - Let's try with manual setting of dimensions (less flexible, but better if correct) [08:20:55] sure [08:21:02] going to revert the hack then [08:21:23] Sure elukey [08:21:54] elukey: The same thin happen with pageview_[hourly|daily] actually, but with a dimension [08:21:58] elukey: :( [08:26:46] :( [08:55:49] !log Kill - Restart mediawiki-history-druid-coord to pick last update [08:55:50] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:32:48] 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3576622 (10elukey) I would set log_retention_bytes via puppet (maximum size of the topic partitions) to limit the size of /var/spool/kafka. [09:51:07] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Set up LVS and VirtualHost for RunSingleJob.php - https://phabricator.wikimedia.org/T174599#3576647 (10mobrovac) > set up a separate VirtualHost for the RunSingleJob.php This is not strictly needed, as LVS will run on top of that,... [09:51:42] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Set up LVS and VirtualHost for RunSingleJob.php - https://phabricator.wikimedia.org/T174599#3576649 (10mobrovac) [09:55:14] 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3576663 (10elukey) ``` du -hsc /var/spool/kafka [..] 47M webrequest_text-1 47M webrequest_text-10 47M webrequest_text-12 47M webrequest_text-13 47M webrequest_text-14 47M webrequest_text-... [10:01:45] 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3576670 (10elukey) Added a puppet prefix `deployment-kafka` in Horizon and added `confluent::kafka::broker::log_retention_bytes: 200000000`, let's see how it goes when applied. [10:02:08] elukey: That's weird that consumer offset get so big [10:02:18] elukey: Or so I think [10:03:36] joal: it is eventlogging related, and the retention time is 48h, might get bit with a lot of data (but I am completely ignorant about deployment-prep kafka setup). Plus I have no idea if the retention bytes applies to the consumer offset [10:03:57] hm [10:10:46] joal: pivot broken with the new config [10:11:07] elukey: can I have a look at the logs? [10:11:13] ServerSettings.port must be a number [10:11:28] ahhh no my bad [10:11:28] sorry [10:11:49] pasted the wrong thing [10:11:50] ufff [10:12:02] no prob elukey [10:12:06] :q [10:17:39] joal: done [10:17:57] I hope I copied/pasted correctly [10:18:02] but banner activity doesn't seem to work [10:18:20] ah sure I missed one paste [10:19:10] now it looks better [10:20:05] elukey: Looks great for banners !!! [10:20:24] However for pageviews, while the dimension is in, druid doesn't to work with :( [10:21:01] which one? hourly/daiy? [10:21:20] elukey: I have found the thing, it's a bug at load time ! [10:21:46] elukey: So the patch I did is not needed for pageviews [10:24:13] nice! [10:25:20] elukey: patch provided for pivot for banners, will do another for pageivews (on refinery) [10:28:27] (03PS1) 10Joal: Correct oozie jobs loading pageviews in druid [analytics/refinery] - 10https://gerrit.wikimedia.org/r/375766 (https://phabricator.wikimedia.org/T161824) [10:28:37] elukey: if you have a minute --^ [10:28:58] elukey: Thanks for the quick test and merge on puppet :) [10:29:31] (03CR) 10Elukey: [C: 031] "lgtm" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/375766 (https://phabricator.wikimedia.org/T161824) (owner: 10Joal) [10:29:42] Thanks again elukey :) [10:29:46] joal: you are live on thorium with your patch :) [10:29:52] You rok :) [10:51:49] 10Analytics-Kanban, 10Patch-For-Review: Add zero carrier to pageview_hourly data on druid - https://phabricator.wikimedia.org/T161824#3576878 (10JAllemandou) [10:51:51] 10Analytics-Kanban: Check how pivot updates schema (or maybe make schema explicit on pivot) - https://phabricator.wikimedia.org/T163697#3576877 (10JAllemandou) 05Open>03declined [10:51:53] 10Analytics: upgrade druid and pivot - https://phabricator.wikimedia.org/T157977#3576879 (10JAllemandou) [10:52:16] 10Analytics-Kanban: Check how pivot updates schema (or maybe make schema explicit on pivot) - https://phabricator.wikimedia.org/T163697#3206394 (10JAllemandou) We won't upgrade pivot. [10:56:24] 10Analytics-Kanban: Productionize mediawiki-history-reduced druid ingestion - https://phabricator.wikimedia.org/T174915#3576900 (10JAllemandou) [10:56:41] 10Analytics-Kanban: Productionize mediawiki-history-reduced druid ingestion - https://phabricator.wikimedia.org/T174915#3576912 (10JAllemandou) a:03JAllemandou [11:02:50] * elukey lunch! [11:06:20] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3576962 (10GoranSMilovanovic) [11:06:39] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3576976 (10GoranSMilovanovic) [11:07:36] 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3576977 (10elukey) Another interesting thing is that the following keeps getting logged: ``` [2017-09-04 11:06:03,425] ERROR Processor got uncaught exception. (kafka.network.Processor) jav... [11:10:43] Leaving for a doctor appointment [11:38:28] 10Analytics, 10Beta-Cluster-Infrastructure: deployment-kafka01 - disk is full - https://phabricator.wikimedia.org/T174742#3577006 (10elukey) The responsible is deployment-eventlog02.deployment-prep.eqiad.wmflabs, I stopped/started eventlogging on it (via eventlogctl) and I saw the error msg stopped/started acc... [12:04:39] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Have "Last Attracted Developers" information for Gerrit (already exists for Git) - https://phabricator.wikimedia.org/T151161#3577176 (10Aklapper) >>! In T151161#3576224, @jgbarah wrote: > Not by default, but I did a run of the scripts, s... [12:06:17] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2 bugs (4/4) - Detail page - https://phabricator.wikimedia.org/T170940#3577180 (10fdans) [12:08:24] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3577182 (10Aklapper) a:05Aklapper>03None See https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Requesting_a_Space for required information, e.g. who e... [12:19:27] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3577233 (10GoranSMilovanovic) @JAllemandou In the [[ https://github.com/wikimedia/analytics-refinery/blob/master/oozie/banner_activity/druid/daily... [12:31:33] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3577285 (10GoranSMilovanovic) @JAllemandou Also, I am interested to learn about the data retention lifetime in Druid. For example, the WMDE Summer... [12:32:14] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3577286 (10Addshore) So the line linked to is only an example of a parameter being passed in. You need to have a look at "oozie/banner_activity/dr... [12:37:44] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3577296 (10GoranSMilovanovic) @Addshore Understood: so, the Oozie coordinators are actually responsible for setting the script parameters like des... [13:06:25] 10Analytics, 10Analytics-Cluster, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install new kafka nodes kafka-jumbo100[1-6] - https://phabricator.wikimedia.org/T167992#3577404 (10elukey) So we solved the issue with partman and I was able to install the os on kafka-jumbo100[12], but failed to PXE boot... [13:07:36] joal, HaeB - I checked the eventlogginc_sync logs on dbstore1002 and there is nothing weird going on about Popups_16364296, not sure what's happening [13:08:07] the eventlogginc_sync.log file is not rotated though and its size is now 22G, that makes it difficult to find anything specific to the 30th [13:23:22] 10Analytics, 10Analytics-Cluster, 10WMDE-Analytics-Engineering: Install R on stat1004? - https://phabricator.wikimedia.org/T174890#3577446 (10Addshore) So both stats 1004 and 1005 have the role analytics_cluster::client. 1005 also has statistics::private profile::statistics::private includes ::statistics::co... [13:33:15] ah this is interesting, I just checked and I can see a huge INSERT IGNORE INTO `Popups_16364296` VALUES on db1047 [13:41:20] and now it stops at august 31st [13:42:36] 10Analytics, 10Analytics-EventLogging: Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3573902 (10elukey) On db1047 (analytics-slave) I've ran the query in the description and found: ``` [..] | 2017-08-30... [13:51:55] last log [13:51:56] 2017-09-04T13:49:14 localhost log Popups_11625443 (no new data on master in last 90 days, skipping) [14:11:44] just completed the zookeeper restarts on conf100* [14:25:27] 10Analytics, 10EventBus, 10Operations, 10User-Elukey: Eventbus does not handle gracefully changes in DNS recursors - https://phabricator.wikimedia.org/T171048#3577723 (10elukey) a:05elukey>03None [14:44:25] 10Analytics, 10Analytics-Wikistats: Provide yearly update of stats for audit report - https://phabricator.wikimedia.org/T174950#3577791 (10Erik_Zachte) [14:47:30] 10Analytics-Kanban, 10Analytics-Wikistats: Use daily granularity for 1-month time ranges - https://phabricator.wikimedia.org/T173372#3577832 (10fdans) [14:49:49] 10Analytics, 10Analytics-Wikistats: Provide yearly update of stats for audit report - https://phabricator.wikimedia.org/T174950#3577860 (10Erik_Zachte) After a long outage (partly caused by unexpected server migration) I revived Wikistats scripts and parsed dumps in the last 10 days. So I can partially answer... [15:00:25] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3577879 (10JAllemandou) @GoranSMilovanovic : The temporary location is deleted as part of the oozie workflow (https://github.com/wikimedia/analyti... [15:03:05] 10Analytics, 10Analytics-Wikistats: Provide yearly update of stats for audit report - https://phabricator.wikimedia.org/T174950#3577917 (10Erik_Zachte) As for data that Wikistats can supply: (1/2) For the year ended June 30, 2017, volunteers added approximately 7.5 million images, movies, and sound files to t... [15:19:53] Thanks again a-team for the nice moment with Lino :) He's very happy :) [15:20:24] joal: you guys are the best :D [15:20:31] oh my nick [15:22:11] \o/ [15:22:33] ah I forgot to show you guys the eventlogging_cleaner.py stuff [15:22:37] nevermind, next time [16:03:41] 10Analytics: R execution on stat1005 -> 'stack smashing error' - https://phabricator.wikimedia.org/T174946#3578004 (10Reedy) [16:06:28] * elukey off [16:06:38] talk with you tomorrow team! [16:25:04] (03PS14) 10Joal: Add tranquility to the banner streaming job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/373030 (https://phabricator.wikimedia.org/T168550) [16:27:27] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Check detached accounts in DB with same username for "mediawiki" and "phab" sources but different uuid's (and merge if connected) - https://phabricator.wikimedia.org/T170091#3578082 (10Aklapper) 05Open>03stalled Stalled on an updated... [16:55:32] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Investigate detached duplicated accounts in DB with same username, same source, but different uuids - https://phabricator.wikimedia.org/T170093#3578142 (10Aklapper) 05Open>03stalled [16:55:55] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jul-Sep 2017): Investigate detached duplicated accounts in DB with same username, same source, but different uuids - https://phabricator.wikimedia.org/T170093#3419099 (10Aklapper) p:05Lowest>03Low [17:09:34] 10Analytics, 10Project-Admins, 10Research, 10cloud-services-team: Create a phabricator project called "wikireplica-datasets" - https://phabricator.wikimedia.org/T173512#3578176 (10Aklapper) @Halfak: Could you please answer the last comment? [18:09:18] 10Analytics, 10Analytics-Wikistats: Provide yearly update of stats for audit report - https://phabricator.wikimedia.org/T174950#3578211 (10Erik_Zachte) As for data that Wikistats can supply: (2/2) "For the year ended June 30, 2017, the educational content of the Foundation’s largest project, Wikipedia, grew b... [18:39:45] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578223 (10GoranSMilovanovic) @JAllemandou Thanks. I've already read the Oozie page. As @Addshore would put it, in theory I got it, doing it is mo... [18:48:22] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578243 (10JAllemandou) @GoranSMilovanovic: Retention in Druid for banners is `forever` (we keep all history), but there is a glitch: after 2 mont... [19:15:46] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578287 (10GoranSMilovanovic) @JAllemandou Is that what prevents me to find any wdmesummer campaign when I try to filter by campaign in Pivot? [19:24:15] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 3 others: Visualize page create events for all wikis - https://phabricator.wikimedia.org/T170850#3578310 (10kaldari) Looks like the reports are successfully being populated at https://analytics.wikimedia.org/datasets/periodic/r... [19:52:09] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578360 (10JAllemandou) @GoranSMilovanovic : It wouldn't. I have found this: https://pivot.wikimedia.org/#banner_activity_minutely/line-chart/2/EQ... [20:00:06] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578376 (10GoranSMilovanovic) @JAllemandou That's right. I'm puzzled: today, I was able to find only one WMDE campaign (some test thing) there. H... [20:06:55] Ther used to be active editors in the reportcards graphs, but that is not updated anymore. Can one find it somewhere else? [20:07:59] I had this old URL saved: http://reportcard.wmflabs.org/graphs/active_editors [20:09:39] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578388 (10JAllemandou) @GoranSMilovanovic : A value is `null` (in our case banner) when not set. Given how we generate data, it means the url par... [20:12:52] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578392 (10GoranSMilovanovic) @JAllemandou Thank you very much. @Addshore Do you have any idea of what could have caused the above described beha... [20:32:27] (03PS6) 10Joal: Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) [20:42:01] 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Understand the Webrequest (HDFS) to Druid Mapping - https://phabricator.wikimedia.org/T174917#3578457 (10GoranSMilovanovic) @JAllemandou @Addshore Here's the description of the discrepancy found during the WMDE Summer Banner Campaign 2017:... [21:48:33] 10Analytics, 10Analytics-EventLogging: Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana - https://phabricator.wikimedia.org/T174815#3578511 (10Tbayer) >>! In T174815#3574066, @JAllemandou wrote: ... > > Looks like we still have replication problems @...