[00:05:33] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:10:56] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:11:18] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:17:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:20:15] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [00:21:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0] [00:35:56] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:41:20] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:42:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [00:45:14] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [00:46:59] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [00:47:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:53:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:55:31] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [00:56:28] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:56:44] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:57:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:01:54] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:04:44] 3Continuous-Integration: Jenkins: Figure out long term solution for /tmp management - https://phabricator.wikimedia.org/T74011#950120 (10Krinkle) [01:04:45] 3Continuous-Integration: /tmp/MWDocGen-* files are left behind on Jenkins slaves - https://phabricator.wikimedia.org/T84973#950118 (10Krinkle) 5Open>3stalled p:5Triage>3Normal [01:09:45] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:13:49] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:17:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [01:19:39] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:21:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:21:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:23:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [01:26:31] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<20.00%) [01:27:37] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [01:31:53] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [01:34:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:37:41] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:37:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:38:49] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:39:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:41:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:49:39] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [01:49:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:54:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:55:13] Project beta-scap-eqiad build #36198: FAILURE in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36198/ [02:02:43] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:02:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [02:04:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [02:05:34] Project beta-scap-eqiad build #36199: STILL FAILING in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36199/ [02:06:46] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:49] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:51] Yippee, build fixed! [02:15:51] Project beta-scap-eqiad build #36200: FIXED in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36200/ [02:24:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [02:30:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:42:39] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:58:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:00:45] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:07:41] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [03:09:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:20:39] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:26:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:28:38] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [03:31:42] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:33:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [03:34:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:36:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:37:35] Yippee, build fixed! [03:37:35] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #169: FIXED in 35 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/169/ [03:45:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [03:48:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:50:45] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:51:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [03:56:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [03:56:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:57:57] Yippee, build fixed! [03:57:57] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #388: FIXED in 7 min 20 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/388/ [03:58:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:59:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:01:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:02:54] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:07:48] Yippee, build fixed! [04:07:49] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #448: FIXED in 49 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/448/ [04:09:20] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #277: FAILURE in 9 min 55 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/277/ [04:13:49] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:15:38] Project beta-scap-eqiad build #36213: FAILURE in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36213/ [04:18:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:19:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:20:45] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:21:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [04:23:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [04:26:02] Yippee, build fixed! [04:26:02] Project beta-scap-eqiad build #36214: FIXED in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36214/ [04:28:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:29:20] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:32:15] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #393: FAILURE in 6 min 38 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/393/ [04:32:55] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [04:37:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:38:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:41:14] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:41:40] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:43:29] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #207: FAILURE in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/207/ [04:43:48] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:44:36] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [04:52:17] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:52:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:52:44] Yippee, build fixed! [04:52:44] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #415: FIXED in 44 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/415/ [04:52:45] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [04:58:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [04:59:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:01:44] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #501: FAILURE in 52 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/501/ [05:03:45] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:06:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [05:06:31] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:06:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [05:09:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:17:45] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:20:39] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:22:21] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:22:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:22:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [05:30:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:35:02] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [05:36:28] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:45:35] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [05:53:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:55:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:14:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:21:54] Yippee, build fixed! [06:21:54] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #378: FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/378/ [06:23:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [06:35:13] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:36:31] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:44:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [06:47:31] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:50:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:02:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:04:46] Yippee, build fixed! [07:04:46] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #371: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/371/ [07:05:13] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:11:30] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #377: FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/377/ [07:12:31] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:12:41] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:15:58] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:37:43] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:47:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [08:02:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:05:55] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [08:13:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:14:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:14:49] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:15:12] Project beta-scap-eqiad build #36237: FAILURE in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36237/ [08:18:43] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:19:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [08:25:12] Project beta-scap-eqiad build #36238: STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36238/ [08:27:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [08:31:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:35:27] Yippee, build fixed! [08:35:27] Project beta-scap-eqiad build #36239: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36239/ [08:38:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:41:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:43:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [08:44:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [08:44:47] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:46:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:48:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [08:55:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:56:20] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:58:31] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:59:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:08:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [09:08:43] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:14:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:16:58] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [09:23:37] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:26:36] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [09:27:50] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:29:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:43] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:39:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:39:45] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:40:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:41:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:44:39] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:45:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [09:48:01] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:48:33] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:49:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:52:05] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:55:47] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:57:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:59:27] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:04:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:06:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:10:40] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:17:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [10:17:49] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:18:02] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:34] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [10:20:49] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [10:23:55] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:26:11] Project beta-scap-eqiad build #36250: FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36250/ [10:27:11] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:30] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:28:50] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:29:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:29:42] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:31:44] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:32:32] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:36:16] Yippee, build fixed! [10:36:16] Project beta-scap-eqiad build #36251: FIXED in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36251/ [10:36:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:40:41] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:40:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:42:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [10:47:35] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [10:49:33] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:49:45] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:52:26] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:53:34] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:53:54] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [10:53:58] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [10:56:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:58:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:58:41] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [10:58:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:00:41] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:01:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [11:01:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:05:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:16] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [11:13:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:13:47] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [11:17:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:19:32] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [11:23:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [11:25:42] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:26:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [11:28:12] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [11:28:42] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:29:48] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:31:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:32:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:33:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:33:16] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [11:39:41] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:43:35] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:46:15] Project beta-scap-eqiad build #36258: FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36258/ [11:49:13] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:49:51] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [11:50:43] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:56:10] Project beta-scap-eqiad build #36259: STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36259/ [11:59:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:59:50] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:01:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:02:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:03:03] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:05:41] Yippee, build fixed! [12:05:42] Project beta-scap-eqiad build #36260: FIXED in 1 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36260/ [12:08:38] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [12:10:35] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:12:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:13:25] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:14:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:14:13] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [12:14:53] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [12:15:24] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:15:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:17:19] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:20:12] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [12:20:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [12:22:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [12:22:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:23:42] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:24:12] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [12:25:45] Project beta-scap-eqiad build #36262: FAILURE in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36262/ [12:27:47] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:33:03] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:35:26] Yippee, build fixed! [12:35:26] Project beta-scap-eqiad build #36263: FIXED in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36263/ [12:38:28] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:39:02] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:40:26] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:42:45] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:45:10] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:45:23] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [12:47:40] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [12:48:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:49:28] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:49:58] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:50:48] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:53:19] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:53:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:54:47] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [12:56:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:57:45] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:57:59] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:00:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:00:35] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [13:05:48] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:08:14] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:10:16] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:13:33] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:14:31] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:14:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [13:15:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:18:19] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:21:39] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:25:09] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [13:25:57] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:33:19] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:38:42] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:39:42] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:40:49] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:40:49] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [13:41:51] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:45:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [13:46:55] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:54:43] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:57:41] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:58:40] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:00:58] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [14:01:50] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:04:33] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:05:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:06:11] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:11:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:16:56] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:17:40] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:19:46] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:22:42] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:14] Project beta-scap-eqiad build #36273: FAILURE in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36273/ [14:28:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:08] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:31:32] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:34:18] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:34:32] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:35:03] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:35:19] Project beta-scap-eqiad build #36274: STILL FAILING in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36274/ [14:35:29] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:36:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [14:36:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:39:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:45:18] Project beta-scap-eqiad build #36275: STILL FAILING in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36275/ [14:45:31] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:53:36] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:56:03] Project beta-scap-eqiad build #36276: STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36276/ [14:59:20] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:00:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:01:16] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:01:22] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [15:01:28] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:05:16] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:05:37] Project beta-scap-eqiad build #36277: STILL FAILING in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36277/ [15:06:43] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:43] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:08:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:09:36] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [15:10:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:11:50] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:15:36] Yippee, build fixed! [15:15:37] Project beta-scap-eqiad build #36278: FIXED in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36278/ [15:16:31] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [15:18:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:19:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:20:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:21:23] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:26:13] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:26:23] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:45] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:11] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:33] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:37:23] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:38:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:40:42] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:41:50] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:49:15] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:49:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:49:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:50:43] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:56:37] Project beta-scap-eqiad build #36282: FAILURE in 2 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36282/ [15:56:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:00:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:01:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:03:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:06] Yippee, build fixed! [16:06:07] Project beta-scap-eqiad build #36283: FIXED in 2 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36283/ [16:06:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:07:25] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:07:55] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:10:41] PROBLEM - Puppet failure on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:11:59] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:14:15] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:06] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:17:12] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:19:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:19:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:25:01] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:29:30] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:31:33] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:37:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [16:37:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [16:37:55] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:40:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:40:42] RECOVERY - Puppet failure on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:44] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:41:04] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:44:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:47:17] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [16:48:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:48:46] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:50:04] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:50:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:45] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:59:29] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:03:38] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [17:05:38] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:18] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:06:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [17:18:45] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:41] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:55] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:31:12] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:44] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:37:54] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:37:58] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:36] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:41:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:45:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:30] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:48:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:49:49] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:51:46] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:55:42] Project beta-scap-eqiad build #36294: FAILURE in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36294/ [17:58:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [17:58:25] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:58:55] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:07] Project beta-scap-eqiad build #36295: STILL FAILING in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36295/ [18:06:44] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:06:58] bd808: theoretically there is a Scrum of Scrums today, I'll mention the beta filesystem downtime there. [18:07:37] *nod* Coren may or not hear things via that channel :) [18:08:03] I'm not sure if it will blow up any worse than image uploads not working [18:08:14] That will be broken for sure [18:08:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:08:45] bd808: Andrew is real good about getting Ops concerns handled [18:09:21] very true [18:09:56] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:12:30] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:15:40] Yippee, build fixed! [18:15:40] Project beta-scap-eqiad build #36296: FIXED in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36296/ [18:18:39] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [18:19:46] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:21:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:22:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:22:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [18:23:13] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:23:23] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [18:24:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [18:26:37] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [18:27:08] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:29:39] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:32:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:33:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:34:54] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:43:34] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #224: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/224/ [18:50:24] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:52:08] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:42] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:59:40] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [19:00:53] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #388: FAILURE in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/388/ [19:02:24] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:04:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:08:29] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #449: FAILURE in 50 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/449/ [19:11:06] Yippee, build fixed! [19:11:06] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #278: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/278/ [19:15:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:17:37] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [19:25:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:13] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:27:41] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:22] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:32:48] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:34:36] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:58] Yippee, build fixed! [19:35:58] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #394: FIXED in 9 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/394/ [19:45:29] Project beta-scap-eqiad build #36304: FAILURE in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36304/ [19:47:38] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [19:55:25] Yippee, build fixed! [19:55:26] Project beta-scap-eqiad build #36305: FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36305/ [19:56:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:57:13] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [20:05:06] Yippee, build fixed! [20:05:06] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #502: FIXED in 53 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/502/ [20:05:26] Project beta-scap-eqiad build #36306: FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36306/ [20:08:32] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:15:28] Yippee, build fixed! [20:15:29] Project beta-scap-eqiad build #36307: FIXED in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36307/ [20:20:57] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #427: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/427/ [20:22:46] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:45] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:33:57] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:38:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:41:28] Yippee, build fixed! [20:41:29] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #459: FIXED in 49 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/459/ [20:49:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:50:40] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:55:48] Project beta-scap-eqiad build #36311: FAILURE in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36311/ [20:58:37] PROBLEM - Puppet failure on deployment-db2 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:58:57] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:44] (03PS1) 10BryanDavis: Add a Composer package definition [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/182379 [21:03:50] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:05:04] Project beta-scap-eqiad build #36312: STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36312/ [21:05:59] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #367: FAILURE in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/367/ [21:08:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [21:13:31] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:13:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [21:15:26] Yippee, build fixed! [21:15:26] Project beta-scap-eqiad build #36313: FIXED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/36313/ [21:15:43] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [21:16:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:19:13] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:19:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:23:11] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:28:40] RECOVERY - Puppet failure on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [21:36:12] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:41:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:14] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:54] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:56] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:51:18] Yippee, build fixed! [21:51:18] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #223: FIXED in 42 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/223/ [21:51:46] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:53:14] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:54:42] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:04:55] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:06:15] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:11:49] Yippee, build fixed! [22:11:49] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #378: FIXED in 28 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/378/ [22:15:55] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:16:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:24:42] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [22:29:54] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [22:30:00] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:32:45] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:32:45] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [22:36:27] Hi chrismcmahon... quick question: can I just add a "platform: android" line to jjb/browsertests.yaml, and have it run on android? Thanks in advance! [22:36:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:38:48] AndyRussG: no, we're only set up to run tests on browsers. we need a whole other framework (e.g. "Appium") to run on devices, and we haven't built that [22:39:34] AndyRussG: file a Phab ticket for that? it's something I'd like to make happen. [22:40:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0] [22:43:32] chrismcmahon: Hmmm... does that mean that the other platform: properties in there (like platform: 'windows_7' and platform: 'os_x_10.9') don't actually do anything? [22:44:28] AndyRussG: those platforms are implemented, but 'android' is not. devices themselves need a different driver. [22:47:50] chrismcmahon: iiinteresting, huh.... where do the details of what platforms are implemented and what are not live? I think someone may have already pointed me to this before, BTW, apologies if that's the case 8p [22:49:12] AndyRussG: in jenkins job builder config [22:49:13] also, why isn't android a platform? I mean, it's an OS, right? And iOS? [22:50:27] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:50:32] AndyRussG: maybe look over https://saucelabs.com/features to see what we have available. [22:57:43] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:02:08] chrismcmahon: https://saucelabs.com/platforms ? [23:05:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [23:06:40] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:08:42] AndyRussG: it might work. I know we've never tried it. [23:09:50] chrismcmahon: hmmm, OK! we're more than happy to be guinnea pigs with CentralNotice :) [23:10:58] chrismcmahon: I'm getting ahead of myself a little bit here, since actual mobile tests for CentralNotice are blocked on something else (some minor changes that need to be deployed to production, or a least the beta cluster) [23:11:28] But as soon as that is solved I'll submit a patch for android and ios platforms, and we can see what happens, sound good? [23:12:21] AndyRussG: I still think we're going to have to build more framework than we have right now, but it's something I'd like to work on. [23:12:46] OK, yeah that sounds great [23:13:28] Other than the above ^ please let me know if there's anything specific I can do? [23:13:52] AndyRussG: do please file a Phab request for Quality Assurance outlining what you want to accomplish here. [23:14:07] chrismcmahon: OK, you bet :) [23:14:34] AndyRussG: if what you [23:14:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [23:15:13] AndyRussG: if what you're actually after is to test the web interface on a particular device, we have done that by spoofing the Useragent in Firefox [23:15:29] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:15:44] AndyRussG: but I *think* you want to drive native apps, not the web interface [23:17:27] chrismcmahon: hmmm ... we don't have native apps, just the web interface, but it would be nice to test that on native android and iOS browsers... [23:17:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:17:58] though spoofing the useragent would be a very nice intermediate and useful step I think [23:18:31] yeah a large number of mobile-related bugs can be reproduced on desktop just spoofing the UA [23:21:27] (^ by "we", I mean CentralNotice, which is the bit of software that I'm working on) [23:21:47] chrismcmahon: how do I set a test to do the UA spoof? [23:22:28] AndyRussG: let me look that up, one moment... [23:22:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:25:48] AndyRussG: see lines 55 and 75: https://github.com/wikimedia/mediawiki-extensions-MobileFrontend/blob/master/tests/browser/features/step_definitions/common_steps.rb [23:26:36] AndyRussG: it's really clunky right now, I had a patch working to extract that more into common code that Dan Duvall will be picking up in the new year [23:27:42] chrismcmahon: fantastic, thanks!! [23:28:06] still looks like a fine start [23:29:15] I'll dig into that and be sure to add you as a reviewer on Gerrit [23:29:38] AndyRussG: another example from Flow: https://github.com/wikimedia/mediawiki-extensions-Flow/blob/master/tests/browser/features/step_definitions/flow_no_javascript_steps.rb [23:29:46] we really need to make this common code [23:31:50] AndyRussG: thanks for pushing the framework :-) much of this work is falling now to Dan Duvall, as it gets more complex [23:40:59] AndyRussG: just FYI, that useragent hack only works for Firefox, no other browser supports it [23:43:05] chrismcmahon: got it [23:46:31] PROBLEM - Free space - all mounts on deployment-cache-upload02 is CRITICAL: CRITICAL: deployment-prep.deployment-cache-upload02.diskspace._srv_vdb.byte_percentfree.value (<100.00%) [23:47:47] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:51:04] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:51:28] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:52:04] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:53:44] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]