[00:16:24] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [00:33:39] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:41:25] RECOVERY - Puppet failure on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:51] (03PS1) 10Unicodesnowman: Make jslint voting for VipsScaler [integration/config] - 10https://gerrit.wikimedia.org/r/181948 [03:28:38] (03CR) 10Gergő Tisza: [C: 031] Make jslint voting for VipsScaler [integration/config] - 10https://gerrit.wikimedia.org/r/181948 (owner: 10Unicodesnowman) [04:05:15] Yippee, build fixed! [04:05:15] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #442: FIXED in 48 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/442/ [04:14:56] Project beta-scap-eqiad build #35787: FAILURE in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35787/ [04:24:58] Project beta-scap-eqiad build #35788: STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35788/ [04:33:19] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #201: FAILURE in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/201/ [04:35:22] Yippee, build fixed! [04:35:22] Project beta-scap-eqiad build #35789: FIXED in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35789/ [04:56:02] Yippee, build fixed! [04:56:03] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #495: FIXED in 50 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/495/ [05:02:09] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #416: STILL FAILING in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/416/ [05:27:56] Project browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce build #374: STILL FAILING in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-test2.m.wikipedia.org-linux-firefox-sauce/374/ [05:29:33] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #452: STILL FAILING in 47 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/452/ [05:39:45] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce build #211: STILL FAILING in 55 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-windows_8-internet_explorer-sauce/211/ [06:37:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:53:51] Yippee, build fixed! [06:53:51] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #365: FIXED in 14 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/365/ [06:54:41] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:07:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [07:19:39] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [10:16:59] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:34:59] Project beta-scap-eqiad build #35825: FAILURE in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35825/ [10:45:27] Project beta-scap-eqiad build #35826: STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35826/ [10:55:33] Yippee, build fixed! [10:55:33] Project beta-scap-eqiad build #35827: FIXED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35827/ [11:00:45] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [11:01:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [11:30:46] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:47] PROBLEM - Puppet failure on deployment-cache-bits01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:04:32] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:06:12] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:15:05] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:29:35] RECOVERY - Puppet failure on deployment-cache-text02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:31:13] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:31:39] PROBLEM - Puppet failure on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:39:39] PROBLEM - Puppet failure on deployment-rsync01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:39:45] RECOVERY - Puppet failure on deployment-cache-bits01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:40:03] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [13:04:41] RECOVERY - Puppet failure on deployment-rsync01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:12:44] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:17:57] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:37:43] RECOVERY - Puppet failure on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:42:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:09] PROBLEM - Puppet failure on deployment-cache-mobile03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:46:21] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:09:23] Project beta-scap-eqiad build #35862: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35862/ [17:20:58] Yippee, build fixed! [17:20:58] Project beta-scap-eqiad build #35863: FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35863/ [17:58:55] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:23:57] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [18:35:13] Project beta-scap-eqiad build #35872: FAILURE in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35872/ [18:37:43] PROBLEM - Puppet failure on deployment-cache-upload02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:45:19] Yippee, build fixed! [18:45:19] Project beta-scap-eqiad build #35873: FIXED in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35873/ [18:52:51] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #356: FAILURE in 1 min 24 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/356/ [19:01:42] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:06:35] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #443: FAILURE in 48 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/443/ [19:07:43] RECOVERY - Puppet failure on deployment-cache-upload02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:15:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:16:02] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [19:26:43] RECOVERY - Puppet failure on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:37:53] Yippee, build fixed! [19:37:53] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #202: FIXED in 43 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/202/ [19:40:41] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [19:41:05] RECOVERY - Puppet failure on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:49:03] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #410: FAILURE in 42 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/410/ [19:58:34] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #496: FAILURE in 51 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/496/ [20:11:40] PROBLEM - Puppet failure on deployment-bastion is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:41:40] RECOVERY - Puppet failure on deployment-bastion is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:42] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:58:41] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:18:42] RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0] [21:25:03] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #371: FAILURE in 3 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/371/ [21:28:29] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:28:43] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [21:39:56] PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:43:24] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce build #217: FAILURE in 39 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-9-sauce/217/ [21:44:29] !log Restarted logstash because redis input was not being processed [21:45:15] Project beta-scap-eqiad build #35891: FAILURE in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35891/ [21:52:44] !log Running apt-get dist-upgrade on deployment-mediawiki03 after seeing lots of hhvm core dumps in logstash and noticing that hhvm and other packages there were out of date. [21:53:29] RECOVERY - Puppet failure on deployment-memc03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:55:14] Yippee, build fixed! [21:55:14] Project beta-scap-eqiad build #35892: FIXED in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35892/ [21:57:44] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #366: FAILURE in 14 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/366/ [22:00:45] Yippee, build fixed! [22:00:45] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #372: FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/372/ [22:04:56] RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [22:05:22] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 48962 bytes in 1.390 second response time [22:12:43] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [23:15:35] Project beta-scap-eqiad build #35900: FAILURE in 1 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35900/ [23:25:00] Project beta-scap-eqiad build #35901: STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35901/ [23:35:35] Yippee, build fixed! [23:35:35] Project beta-scap-eqiad build #35902: FIXED in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/35902/