[00:00:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:00:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [00:06:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [00:07:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [00:12:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:13:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%) [00:14:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:16:13] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:19:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [00:20:02] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [00:25:49] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:28:42] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:29:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:30:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:33:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:33:47] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:33:51] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:36:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:36:26] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:39:47] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:42:10] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:46:17] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [00:46:39] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:47:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:48:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<50.00%) [00:49:17] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:52:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:55:48] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [00:57:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:40] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:50] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:24] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:49] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:07:14] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [01:14:15] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:15:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [01:17:24] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:18:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [01:19:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:28:33] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:37:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:38:41] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [02:08:42] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:09:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [02:13:35] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:22:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:34:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:39:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:50:32] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:58:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:16:17] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:20:33] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [03:23:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:24:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [03:30:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [03:33:34] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [03:35:36] Yippee, build fixed! [03:35:36] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #175: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/175/ [03:41:20] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:41:37] Yippee, build fixed! [03:41:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #229: FIXED in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/229/ [04:00:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:03:32] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:07:36] Yippee, build fixed! [04:07:37] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #454: FIXED in 50 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/454/ [04:24:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:36:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:51:42] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [04:54:35] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:01:44] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [05:08:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [05:19:57] Yippee, build fixed! [05:19:58] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #432: FIXED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/432/ [05:21:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [05:34:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [05:34:55] Project beta-scap-eqiad build #36645: FAILURE in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36645/ [05:44:58] Project beta-scap-eqiad build #36646: STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36646/ [05:55:26] Yippee, build fixed! [05:55:26] Project beta-scap-eqiad build #36647: FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36647/ [05:55:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:25:31] Project beta-scap-eqiad build #36650: FAILURE in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36650/ [06:35:49] Yippee, build fixed! [06:35:50] Project beta-scap-eqiad build #36651: FIXED in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36651/ [06:38:30] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:39:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:45:24] Yippee, build fixed! [06:45:25] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #371: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/371/ [06:45:31] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:45:55] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [06:49:52] Yippee, build fixed! [06:49:52] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #228: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/228/ [07:01:41] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #377: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/377/ [07:04:39] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:09:23] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #383: FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/383/ [07:10:35] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:36:23] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:45:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [07:51:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:52:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:01:24] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [08:05:14] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:21:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:22:44] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [08:35:13] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [08:52:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:56:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:17:24] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:21:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [09:22:28] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:31:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:42:24] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [09:50:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:56:41] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [09:57:02] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:06:35] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:07:27] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:08:25] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:10:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:13:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:20:27] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:31:35] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:35:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:36:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:37:33] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:38:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:43:38] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:58] Project beta-scap-eqiad build #36676: FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36676/ [10:54:49] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:56:15] Yippee, build fixed! [10:56:16] Project beta-scap-eqiad build #36677: FIXED in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36677/ [11:01:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:06:44] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:07:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [11:08:28] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:11:49] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:13:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:18:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:24:46] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:26:39] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [11:33:31] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:36:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:36:44] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:41:45] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:52:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:56:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:57:41] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [11:59:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:02:15] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [12:06:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:06:48] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:08:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:09:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:11:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:11:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [12:12:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:13:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [12:24:39] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:27:15] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [12:29:35] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:31:39] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:36:51] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [12:37:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:38:09] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:38:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [12:39:24] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:41:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:42:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:43:02] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:44:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [12:54:32] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:54:40] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:01:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [13:08:10] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [13:14:45] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:15:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:22:01] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:22:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:27:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:30:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:32:20] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:40:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [13:45:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:45:40] Project beta-scap-eqiad build #36694: FAILURE in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36694/ [13:47:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [13:48:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:51:32] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:52:01] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:52:36] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:55:13] Yippee, build fixed! [13:55:14] Project beta-scap-eqiad build #36695: FIXED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36695/ [13:55:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:01:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:02:19] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:03:40] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:04:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [14:11:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:16:34] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [14:18:44] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:21:38] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:25] 3Continuous-Integration: jenkins on release branch has problem when patch set contains js files - https://phabricator.wikimedia.org/T67085#953240 (10Umherirrender) a:5Umherirrender>3None [14:27:50] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:28:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:45] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:33:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:35:28] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:36:23] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:38:19] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:39:45] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:52] Project beta-scap-eqiad build #36700: FAILURE in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36700/ [14:50:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:56:00] Project beta-scap-eqiad build #36701: STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36701/ [14:57:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:57:51] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [14:57:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:58:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:03:12] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:05:21] Yippee, build fixed! [15:05:21] Project beta-scap-eqiad build #36702: FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36702/ [15:17:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:19:14] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:20:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:22:42] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:37:44] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:42:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:39] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [15:44:13] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:46:13] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:47:35] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:50:01] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:52:23] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:53:43] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:54:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:59:40] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:59:42] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:35] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:06:09] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:07:39] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:45] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:10:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:11:14] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:34] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:15:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:31] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:43] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [16:24:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:46] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:31:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:33:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:36:11] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:47] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:39:31] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:45] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:41:43] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:45:09] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:46:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:49:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:53:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:57:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:58:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:59:49] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:03] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:46] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:50] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:40] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:10:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [17:12:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:14:38] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [17:14:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:16:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:18:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:45] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:23:53] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:31] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:37:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:37] Project beta-scap-eqiad build #36718: FAILURE in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36718/ [17:45:39] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:47:43] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:49:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:49:33] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:46] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:48] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:56:25] Project beta-scap-eqiad build #36719: STILL FAILING in 2 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36719/ [18:01:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:04:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [18:05:09] Project beta-scap-eqiad build #36720: STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36720/ [18:10:42] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:13:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:15:42] Yippee, build fixed! [18:15:43] Project beta-scap-eqiad build #36721: FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36721/ [18:15:48] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:18:35] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #265: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/265/ [18:19:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:26:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:27:13] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:32:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:33:19] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:36:04] Project beta-scap-eqiad build #36723: FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36723/ [18:38:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:39:54] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #176: FAILURE in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/176/ [18:40:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:42:40] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:45:50] Project beta-scap-eqiad build #36724: STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36724/ [18:50:24] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:50:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:52:14] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [18:52:44] Project browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce build #370: FAILURE in 51 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-windows_8-internet_explorer-sauce/370/ [18:54:18] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #455: FAILURE in 35 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/455/ [18:56:03] Yippee, build fixed! [18:56:04] Project beta-scap-eqiad build #36725: FIXED in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36725/ [18:57:25] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:59:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:59:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:00:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [19:02:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:03:19] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:40] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:08:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [19:08:48] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:10:13] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #284: FAILURE in 9 min 54 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/284/ [19:15:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:15:41] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:42] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:24:09] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:21] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #214: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/214/ [19:24:55] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:27] Project beta-scap-eqiad build #36728: FAILURE in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36728/ [19:25:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:30:38] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #422: FAILURE in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/422/ [19:31:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:33:42] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:33:44] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #357: FAILURE in 3 min 6 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/357/ [19:33:52] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:29] Yippee, build fixed! [19:35:30] Project beta-scap-eqiad build #36729: FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36729/ [19:35:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:38:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:40:24] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #265: FAILURE in 8 min 41 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/265/ [19:51:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:54:17] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:54:23] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:55:47] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:58:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:00:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:01:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:04:57] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:06:07] Project beta-scap-eqiad build #36732: FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36732/ [20:06:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [20:14:40] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #508: FAILURE in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/508/ [20:15:24] Yippee, build fixed! [20:15:24] Project beta-scap-eqiad build #36733: FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36733/ [20:15:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:16:44] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:18:40] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:19:18] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:23:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:23:41] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:24:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:25:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [20:26:25] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #465: FAILURE in 51 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/465/ [20:28:06] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #433: FAILURE in 18 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/433/ [20:29:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:29:48] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:34:53] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [20:39:46] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:41:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:48:40] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:55] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #373: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/373/ [20:51:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:53:35] Project beta-scap-eqiad build #36735: FAILURE in 2 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36735/ [20:56:06] Yippee, build fixed! [20:56:07] Project beta-scap-eqiad build #36736: FIXED in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36736/ [20:59:45] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:59:53] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:00:49] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:04:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:05:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [21:06:13] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [21:09:31] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:16:44] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [21:33:47] Project browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce build #372: FAILURE in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-test2.wikipedia.org-linux-chrome-sauce/372/ [21:37:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:39:29] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:40:15] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #229: FAILURE in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/229/ [21:41:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [0.0] [21:50:25] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:52:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:52:42] Yippee, build fixed! [21:52:43] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #378: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/378/ [21:56:01] Project beta-scap-eqiad build #36742: FAILURE in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36742/ [22:01:43] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:02:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [22:05:45] Yippee, build fixed! [22:05:45] Project beta-scap-eqiad build #36743: FIXED in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36743/ [22:06:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [22:11:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:17:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [22:20:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:21:51] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [22:22:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:26:45] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:31:29] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:36:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:46] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:47:13] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:55:28] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:00:43] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:06:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:06:54] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:08:12] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:10:49] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:13:02] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:13:42] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:16:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:17:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:20:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:31:33] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:33:11] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [23:36:57] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:40:47] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:43:03] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:43:43] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [23:47:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [23:54:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]