[00:34:46] Project beta-scap-eqiad build #36047: FAILURE in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36047/ [00:45:10] Project beta-scap-eqiad build #36048: STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36048/ [00:55:34] Yippee, build fixed! [00:55:35] Project beta-scap-eqiad build #36049: FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36049/ [01:07:29] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<33.33%) [02:31:20] (03CR) 10XZise: "Oh I thought the code were Python 3 compatible. With Ie1e6760 the flake83 tests should succeed so it's not necessary to have it non-voting" [integration/config] - 10https://gerrit.wikimedia.org/r/182067 (owner: 10XZise) [03:33:55] Yippee, build fixed! [03:33:55] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #167: FIXED in 31 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/167/ [04:02:38] Yippee, build fixed! [04:02:39] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #446: FIXED in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/446/ [04:45:40] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [04:55:09] Yippee, build fixed! [04:55:10] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #499: FIXED in 51 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/499/ [05:15:40] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:15:55] Yippee, build fixed! [05:15:56] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #424: FIXED in 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/424/ [05:50:24] Yippee, build fixed! [05:50:24] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #364: FIXED in 46 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/364/ [06:33:33] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:37:31] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:52:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:54:19] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #369: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/369/ [06:55:01] Project beta-scap-eqiad build #36084: FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36084/ [07:03:36] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:05:11] Project beta-scap-eqiad build #36085: STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36085/ [07:15:19] Yippee, build fixed! [07:15:20] Project beta-scap-eqiad build #36086: FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36086/ [07:22:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:54:02] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:18:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:19:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:14:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:31:40] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:44:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [12:45:29] Project beta-scap-eqiad build #36118: FAILURE in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36118/ [12:55:27] Yippee, build fixed! [12:55:27] Project beta-scap-eqiad build #36119: FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36119/ [12:57:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [13:15:00] Project beta-scap-eqiad build #36121: FAILURE in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36121/ [13:25:04] Project beta-scap-eqiad build #36122: STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36122/ [13:27:16] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [13:35:05] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:35:16] Yippee, build fixed! [13:35:17] Project beta-scap-eqiad build #36123: FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36123/ [13:38:41] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:47:18] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:48:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [13:49:17] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:49:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:53:42] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:54:32] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [13:55:32] Project beta-scap-eqiad build #36125: FAILURE in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36125/ [14:01:34] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:05:31] Yippee, build fixed! [14:05:32] Project beta-scap-eqiad build #36126: FIXED in 1 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36126/ [14:06:10] Project browsertests-Wikidata-PerformanceTests-linux-firefox-sauce build #102: FAILURE in 9.2 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-PerformanceTests-linux-firefox-sauce/102/ [14:08:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:11:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:12:18] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [14:14:16] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:18:58] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:34] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:42] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:20:02] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:37] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [14:35:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:36:26] Yippee, build fixed! [14:36:26] Project beta-mediawiki-config-update-eqiad build #1698: FIXED in 29 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/1698/ [14:36:27] Project beta-scap-eqiad build #36130: FAILURE in 0.96 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36130/ [14:36:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [14:36:46] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:39:48] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:41:41] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [14:44:41] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:46:03] Yippee, build fixed! [14:46:04] Project beta-scap-eqiad build #36131: FIXED in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36131/ [14:46:39] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:49:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:50:43] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:58:41] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:01:48] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:05:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:10:14] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:11:25] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:11:41] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:16:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:19:42] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:20:44] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:22:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [15:28:41] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:46] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:40:17] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:02] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:34] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:49:49] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:52:31] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:56:27] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:58:13] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:00:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:44] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:00:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:14:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:14:50] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:14] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:16] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:33:27] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:34:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:39:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:44:42] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:08] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:46:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:53:15] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [16:55:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:03:22] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:04:40] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:09:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [17:14:42] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:18:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:20:38] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:30] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:34:23] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:36:22] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:39:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:48:40] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:58] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:56:32] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:59:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:00:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:01:25] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [18:01:42] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [18:04:24] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:21:00] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:24:27] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #168: FAILURE in 22 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/168/ [18:24:37] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [18:30:40] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:35:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:36:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [18:46:15] Project beta-scap-eqiad build #36155: FAILURE in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36155/ [18:49:58] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:50:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:51:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:55:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:55:50] Yippee, build fixed! [18:55:51] Project beta-scap-eqiad build #36156: FIXED in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36156/ [18:57:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:01:20] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #387: FAILURE in 8 min 27 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/387/ [19:05:42] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:06:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:12:39] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #447: FAILURE in 54 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/447/ [19:15:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [19:15:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:24] Project beta-scap-eqiad build #36158: FAILURE in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36158/ [19:17:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:20:45] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:21:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:21:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:22:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:58] Yippee, build fixed! [19:25:59] Project beta-scap-eqiad build #36159: FIXED in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36159/ [19:28:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:28:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:31:44] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:32:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:38:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:40:15] Project beta-scap-eqiad build #36160: FAILURE in 5 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36160/ [19:41:40] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:42:43] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:45:50] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:46:03] Project beta-scap-eqiad build #36161: STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36161/ [19:46:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:56:45] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:29] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [19:58:46] Yippee, build fixed! [19:58:47] Project beta-scap-eqiad build #36162: FIXED in 4 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36162/ [20:03:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [20:05:18] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #414: FAILURE in 52 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/414/ [20:06:40] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [20:13:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:14:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:15:48] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [20:15:52] Project beta-scap-eqiad build #36164: FAILURE in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36164/ [20:17:46] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:24:15] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:29:42] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:31:56] Yippee, build fixed! [20:31:56] Project beta-scap-eqiad build #36165: FIXED in 7 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36165/ [20:38:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:38:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:47:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:47:41] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:49:41] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:50:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:52:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:53:18] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:53:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [20:54:14] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [20:59:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [21:03:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:03:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [21:15:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [21:16:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:16:18] Project beta-scap-eqiad build #36170: FAILURE in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36170/ [21:16:32] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [21:17:24] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:18:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:18:42] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:19:46] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:19:46] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:22:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:23:20] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:26:49] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:27:40] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:28:21] Yippee, build fixed! [21:28:22] Project beta-scap-eqiad build #36171: FIXED in 4 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36171/ [21:30:33] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:30:43] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:31:16] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:31:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:32:08] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:32:25] !log Updated scap to include 9a37ee0 (Add a warning when attempting to sync a symlink) [21:32:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:34:25] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:34:56] !log puppet failures are "Error: /Stage[main]/Ldap::Client::Utils/File[/usr/local/sbin/delete-ldap-group]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/ldap/scripts/delete-ldap-group: getaddrinfo: Name or service not known" [21:35:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:36:36] Project beta-scap-eqiad build #36172: FAILURE in 2 min 39 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36172/ [21:37:30] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:39:41] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:41:32] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:42:24] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [21:42:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [0.0] [21:42:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:42:49] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:43:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:44:00] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:44:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:45:13] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:45:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:47:00] Project beta-scap-eqiad build #36173: STILL FAILING in 3 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36173/ [21:47:38] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #377: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/377/ [21:48:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:49:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:49:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:42] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:52:41] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:52:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:58] shinken is not very helpful [21:55:34] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [21:56:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [21:57:13] Yippee, build fixed! [21:57:14] Project beta-scap-eqiad build #36174: FIXED in 3 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36174/ [21:58:24] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:59:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:00:14] chrismcmahon: I think the labs DNS server is freaking out [22:00:28] All the puppet failures I looked at were dns related [22:00:35] bd808: seems that way. hasn't it been having problems for a while? [22:00:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:01:06] Yeah it has been flaky off and on for months but this seems to be pretty consistent [22:02:01] I don't know that anyone other than Coren, Yuvi or Andrew could do anything about it and I think they are all AFK [22:02:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:03:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:04:42] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:04:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [22:06:06] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:06:38] bd808: I think Coren was talking about scrapping the whole current DNS setup, but I don't know details. [22:07:49] Yea, there's a planning bug at least. [22:08:13] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [22:08:16] I've seem he and Ryan chat about what it would involve a bit. [22:09:04] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:09:58] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:11:34] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:11:50] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:12:33] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:12:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:14:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:15:14] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [22:15:18] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:16:19] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:16:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:17:05] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:17:34] Project beta-scap-eqiad build #36176: FAILURE in 3 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36176/ [22:17:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:18:32] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:20:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:20:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:23:24] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [22:23:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:24:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [22:24:19] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #221: FAILURE in 51 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/221/ [22:27:28] Project beta-scap-eqiad build #36177: STILL FAILING in 3 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36177/ [22:29:08] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:29:28] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:30:25] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:30:33] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:32:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:33:48] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:34:56] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:33] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:35] Project beta-scap-eqiad build #36178: STILL FAILING in 2 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36178/ [22:36:47] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [22:36:51] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:41:44] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:42:32] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:45:44] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:46:14] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:47:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:47:16] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:48:42] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:49:22] Project beta-scap-eqiad build #36179: STILL FAILING in 5 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36179/ [22:50:39] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [22:50:42] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [22:52:33] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:52:49] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:52:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:56:42] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:58:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:58:33] Yippee, build fixed! [22:58:33] Project beta-scap-eqiad build #36180: FIXED in 4 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36180/ [22:59:13] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:59:25] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:00:17] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:00:35] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:00:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [23:03:51] Yippee, build fixed! [23:03:51] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #376: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/376/ [23:04:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:05:54] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:06:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:06:49] Project beta-scap-eqiad build #36181: FAILURE in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36181/ [23:10:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [23:11:28] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:11:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:11:44] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:11:58] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:14:09] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:15:42] Yippee, build fixed! [23:15:42] Project beta-scap-eqiad build #36182: FIXED in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36182/ [23:16:15] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [23:17:35] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [23:17:51] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:18:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:21:42] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:22:48] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:23:06] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [23:23:46] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:24:14] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:24:24] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [23:28:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [23:29:44] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:30:55] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [23:34:29] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:35:48] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:36:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:36:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [23:37:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:37:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [23:40:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:42:07] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:44:10] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [23:45:29] Project beta-scap-eqiad build #36185: FAILURE in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36185/ [23:46:36] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [23:52:18] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:55:39] Yippee, build fixed! [23:55:40] Project beta-scap-eqiad build #36186: FIXED in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36186/