[00:05:21] 3Beta-Cluster: monitor unsigned salt keys - https://phabricator.wikimedia.org/T72862#944148 (10yuvipanda) http://docs.saltstack.com/en/latest/ref/configuration/master.html#auto-accept salt has auto-accept now. [00:06:10] 3Beta-Cluster: monitor unsigned salt keys - https://phabricator.wikimedia.org/T72862#944149 (10yuvipanda) Although not in the version we are running atm.. [01:44:57] Project beta-scap-eqiad build #35506: FAILURE in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35506/ [01:55:25] Yippee, build fixed! [01:55:25] Project beta-scap-eqiad build #35507: FIXED in 1 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35507/ [03:50:53] Yippee, build fixed! [03:50:53] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce build #378: FIXED in 7 min 18 sec: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-chrome-sauce/378/ [04:32:33] Yippee, build fixed! [04:32:33] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #197: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/197/ [04:44:22] Yippee, build fixed! [04:44:22] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #405: FIXED in 41 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/405/ [04:55:42] Yippee, build fixed! [04:55:42] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #491: FIXED in 50 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/491/ [06:39:20] Yippee, build fixed! [06:39:21] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #212: FIXED in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/212/ [06:53:16] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #361: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/361/ [07:55:13] Project beta-scap-eqiad build #35543: FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35543/ [08:05:16] Project beta-scap-eqiad build #35544: STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35544/ [08:15:57] Yippee, build fixed! [08:15:58] Project beta-scap-eqiad build #35545: FIXED in 1 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35545/ [08:51:17] 3Beta-Cluster: monitor unsigned salt keys - https://phabricator.wikimedia.org/T72862#944318 (10yuvipanda) Also how are keys accepted for new labs instances now? @ArielGlenn [12:31:40] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:14:55] Project beta-scap-eqiad build #35579: FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35579/ [14:25:01] Project beta-scap-eqiad build #35580: STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35580/ [14:35:20] Yippee, build fixed! [14:35:21] Project beta-scap-eqiad build #35581: FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35581/ [15:36:01] Project beta-scap-eqiad build #35587: FAILURE in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35587/ [15:45:02] PROBLEM - Puppet failure on deployment-redis02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:45:34] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:45:56] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:46:10] Yippee, build fixed! [15:46:10] Project beta-scap-eqiad build #35588: FIXED in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35588/ [15:47:04] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [15:50:18] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [15:50:48] PROBLEM - Puppet failure on deployment-eventlogging02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:50:48] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:50:58] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:51:17] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:51:31] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:53:21] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:55:48] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:56:42] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:56:52] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:57:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:59:02] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [15:59:40] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:59:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:01:11] PROBLEM - Puppet failure on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:07:16] Project beta-scap-eqiad build #35590: FAILURE in 3 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35590/ [16:10:01] RECOVERY - Puppet failure on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:27] PROBLEM - Puppet failure on deployment-memc02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:14:41] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:15:17] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:33] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:16:25] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:17:20] Yippee, build fixed! [16:17:20] Project beta-scap-eqiad build #35591: FIXED in 3 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35591/ [16:21:18] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:21:44] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:21:44] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:22:00] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [16:24:42] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:24:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [16:26:44] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:40] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:16] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [16:35:45] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:36:26] Project beta-scap-eqiad build #35593: FAILURE in 2 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35593/ [16:36:28] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:37:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:39:27] PROBLEM - Puppet failure on deployment-stream is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:40:33] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:40:51] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:53] RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:57] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:41:05] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:41:26] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:09] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:45:51] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:46:51] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:46:59] Project beta-scap-eqiad build #35594: STILL FAILING in 3 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35594/ [16:47:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:49:13] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:51:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:53:01] PROBLEM - Puppet failure on deployment-upload is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:53:32] PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:56:30] Yippee, build fixed! [16:56:30] Project beta-scap-eqiad build #35595: FIXED in 2 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35595/ [16:56:45] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:57:29] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:58:21] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:14] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:59:44] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:44] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:00:44] RECOVERY - Puppet failure on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:01:16] RECOVERY - Puppet failure on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:02:42] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [17:04:22] RECOVERY - Puppet failure on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:26] Project beta-scap-eqiad build #35596: FAILURE in 2 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35596/ [17:07:01] PROBLEM - Puppet failure on deployment-restbase02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:08:53] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:09:23] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:12:19] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:15:32] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:16:32] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:16:50] RECOVERY - Puppet failure on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [17:16:50] Project beta-scap-eqiad build #35597: STILL FAILING in 2 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35597/ [17:18:34] RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:17] RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:26] Yippee, build fixed! [17:25:26] Project beta-scap-eqiad build #35598: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35598/ [17:25:40] RECOVERY - Puppet failure on deployment-cxserver03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:12] RECOVERY - Puppet failure on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:26:40] PROBLEM - Puppet failure on deployment-restbase03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:34:12] RECOVERY - Puppet failure on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:22] RECOVERY - Puppet failure on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:56] RECOVERY - Puppet failure on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:18] RECOVERY - Puppet failure on deployment-apertium01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:26] RECOVERY - Puppet failure on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:33] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:37:59] RECOVERY - Puppet failure on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:49] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:47] RECOVERY - Puppet failure on deployment-eventlogging02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:55] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:31] RECOVERY - Puppet failure on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:48] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:32] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:42:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:44] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:47:42] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:49:42] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [17:51:34] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:51:40] RECOVERY - Puppet failure on deployment-restbase03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:16:47] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #87: FAILURE in 2 min 15 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/87/ [20:34:57] Project beta-scap-eqiad build #35617: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35617/ [20:36:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:45:06] Project beta-scap-eqiad build #35618: STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35618/ [20:54:46] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #357: FAILURE in 45 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/357/ [20:55:19] Yippee, build fixed! [20:55:20] Project beta-scap-eqiad build #35619: FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35619/ [21:01:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:03] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #213: FAILURE in 41 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/213/ [22:00:29] Yippee, build fixed! [22:00:30] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #362: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/362/ [23:28:28] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:53:31] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0]