[02:00:32] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 1.905 second response time [02:54:57] Project beta-scap-eqiad build #35655: FAILURE in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35655/ [03:05:03] Project beta-scap-eqiad build #35656: STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35656/ [03:15:19] Yippee, build fixed! [03:15:19] Project beta-scap-eqiad build #35657: FIXED in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35657/ [03:22:18] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:11] RECOVERY - App Server Main HTTP Response on deployment-mediawiki02 is OK: HTTP OK: HTTP/1.1 200 OK - 48961 bytes in 0.564 second response time [03:37:42] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #215: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/215/ [04:03:50] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #440: FAILURE in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/440/ [04:10:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:13:51] Yippee, build fixed! [04:13:52] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #88: FIXED in 51 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/88/ [04:30:34] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #199: FAILURE in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/199/ [04:40:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:53:43] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #493: FAILURE in 49 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/493/ [05:47:46] Yippee, build fixed! [05:47:47] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #358: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/358/ [09:14:42] Project beta-scap-eqiad build #35693: FAILURE in 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35693/ [09:24:14] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:26:07] Project beta-scap-eqiad build #35694: STILL FAILING in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35694/ [09:27:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:27:55] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:28:57] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:29:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:29:33] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:29:51] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:31:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:31:42] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:33:28] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:33:28] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:33:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:33:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:36:34] Project beta-scap-eqiad build #35695: STILL FAILING in 2 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35695/ [09:38:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:39:39] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:40:13] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:41:05] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:41:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:41:47] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:42:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:43:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:45:40] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [09:46:01] Yippee, build fixed! [09:46:02] Project beta-scap-eqiad build #35696: FIXED in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35696/ [09:47:14] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [09:47:52] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:50:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:54:08] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:30] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:55:22] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:56:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:56:33] Project beta-scap-eqiad build #35697: FAILURE in 2 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35697/ [09:56:39] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:57:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:57:48] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:57:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:57:59] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:59:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [09:59:49] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:01:43] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [10:01:43] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:03:47] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:05:15] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:05:16] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [10:06:42] Project beta-scap-eqiad build #35698: STILL FAILING in 2 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35698/ [10:07:13] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:37] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:10:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [10:11:05] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:11:43] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:12:47] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:14:47] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:16:34] Yippee, build fixed! [10:16:35] Project beta-scap-eqiad build #35699: FIXED in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35699/ [10:17:42] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:19:32] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:20:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [10:20:39] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:21:43] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:21:55] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:22:49] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:23:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:23:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:10] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:25:26] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [10:26:40] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:26:44] Project beta-scap-eqiad build #35700: FAILURE in 2 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35700/ [10:28:45] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:04] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:30:48] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:31:42] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:32:19] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:32:31] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:32:32] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:32:58] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:34:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:35:29] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:36:46] Project beta-scap-eqiad build #35701: STILL FAILING in 2 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35701/ [10:36:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:37:14] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:38:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:38:48] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:41:16] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [10:42:42] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:42:42] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [10:43:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:37] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [10:46:40] Project beta-scap-eqiad build #35702: STILL FAILING in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35702/ [10:47:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:47:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:12] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:51:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:51:58] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [10:52:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:54:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:54:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:55:09] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:55:15] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [10:56:23] Project beta-scap-eqiad build #35703: STILL FAILING in 2 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35703/ [10:57:42] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:57:50] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [10:58:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:00:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [11:00:48] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:01:12] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [11:02:32] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:00] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:03:48] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:46] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:46] Yippee, build fixed! [11:06:47] Project beta-scap-eqiad build #35704: FIXED in 2 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35704/ [11:07:40] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:07:57] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:11:01] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:13:33] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:14:00] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:17:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:17:44] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:19:14] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:19:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:19:42] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:22:13] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:24:29] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:24:45] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:27:45] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:32:40] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:32:56] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [11:36:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [11:36:31] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:36:41] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:57] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:39:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [11:42:15] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [11:44:00] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [11:44:12] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [11:50:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:15:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:27:30] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:52:29] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:53:34] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:59:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:00:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:18:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:19:45] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:25:00] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:49:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:11:46] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:30:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:36:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:36:41] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:46] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:00:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:01:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [18:25:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [18:32:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:35:03] Yippee, build fixed! [18:35:04] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #162: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/162/ [18:40:37] Yippee, build fixed! [18:40:37] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #216: FIXED in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/216/ [18:51:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:52:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:02:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:45] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:17:43] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:36] Yippee, build fixed! [19:35:37] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #200: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/200/ [20:07:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:07:57] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #364: FAILURE in 14 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/364/ [20:37:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [20:39:58] Yippee, build fixed! [20:39:59] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #215: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/215/ [21:54:49] Project beta-scap-eqiad build #35749: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35749/ [22:04:59] Project beta-scap-eqiad build #35750: STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35750/ [22:15:18] Yippee, build fixed! [22:15:18] Project beta-scap-eqiad build #35751: FIXED in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35751/ [23:46:30] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%)