[08:55:43] <elukey>	 trying to re-run the failed webrequest-text coord, it seems that the generate statistics hive job failed for some data streaming exception from a datanode
[08:55:47] <elukey>	 really weird, no logs
[09:08:07] <elukey>	 going to be back in a bit to check
[09:20:31] <elukey>	 it is refining now.. I think it was only a temporary weirdness
[09:20:51] <elukey>	 given it is saturday I'd skip a in dept investigation and see if it re-occurs :)
[09:21:23] <elukey>	 !log re-run failed webrequest-text 2018-04-13-07 job - temporary failure between Hive and HDFS
[09:21:24] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:44:54] <icinga-wm>	 PROBLEM - Zookeeper node JVM Heap usage on druid1003 is CRITICAL: 0.9501 ge 0.95 https://grafana.wikimedia.org/dashboard/db/zookeeper?refresh=5m&orgId=1&panelId=40&fullscreen
[16:59:08] <icinga-wm>	 RECOVERY - Zookeeper node JVM Heap usage on druid1003 is OK: (C)0.95 ge (W)0.9 ge 0.8969 https://grafana.wikimedia.org/dashboard/db/zookeeper?refresh=5m&orgId=1&panelId=40&fullscreen
[18:34:43] <joal>	 Thanks a lot elukey for having rerun the job :)
[19:32:32] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Beta EventLogging pipeline broken - https://phabricator.wikimedia.org/T220890 (10Gilles)
[20:03:31] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Beta EventLogging pipeline broken - https://phabricator.wikimedia.org/T220890 (10Gilles) Looking at the EventError events coming in on one of the kafka brokers:   ` gilles@deployment-kafka-jumbo-1:~$ kafka-console-consumer --bootstrap-server localhos...
[20:19:40] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Performance-Team: Beta EventLogging pipeline broken - https://phabricator.wikimedia.org/T220890 (10Gilles) 05Open→03Resolved Scap-deploying the latest version of /srv/deployment/eventlogging/analytics and restarting the eventlogging services fixed it:  {F28643953,...
[21:27:01] <wikibugs>	 10Analytics, 10Analytics-EventLogging: Update client-side event validator to support (at least) draft 3 of JSON Schema - https://phabricator.wikimedia.org/T182094 (10Aklapper) (removing #tracking tag as there are no subtasks)
[21:49:20] <wikibugs>	 (03PS1) 10QChris: Add .gitreview [analytics/wmde/NewEditors/wmdeBannerCampaigns_Dashboard] - 10https://gerrit.wikimedia.org/r/503672
[21:49:22] <wikibugs>	 (03CR) 10QChris: [V: 03+2 C: 03+2] Add .gitreview [analytics/wmde/NewEditors/wmdeBannerCampaigns_Dashboard] - 10https://gerrit.wikimedia.org/r/503672 (owner: 10QChris)