[00:09:47] PROBLEM - Puppet staleness on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [43200.0] [01:00:26] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL - Socket timeout after 10 seconds [01:05:14] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47101 bytes in 0.813 second response time [02:35:43] Project browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #574: FAILURE in 2 min 42 sec: https://integration.wikimedia.org/ci/job/browsertests-WikiLove-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/574/ [02:47:12] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: Connection refused [03:46:42] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL - Socket timeout after 10 seconds [03:51:40] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 45556 bytes in 5.275 second response time [04:40:39] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #446: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/446/ [04:41:26] PROBLEM - Puppet failure on deployment-stream is CRITICAL 60.00% of data above the critical threshold [0.0] [04:46:27] 6Release-Engineering, 5Monday Morning SWAT Deployment 2015-05-18: SWAT - https://phabricator.wikimedia.org/T99409#1291492 (10mmodell) [05:01:24] RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0] [05:35:30] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #421: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/421/ [05:44:43] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL - Socket timeout after 10 seconds [05:49:38] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 45566 bytes in 4.392 second response time [06:37:27] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 20.00% of data above the critical threshold [0.0] [06:37:45] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:40:21] RECOVERY - Free space - all mounts on deployment-eventlogging02 is OK All targets OK [06:41:14] 6Release-Engineering, 6Project-Creators: SWAT Project (Tag) - https://phabricator.wikimedia.org/T99411#1291507 (10mmodell) 3NEW [07:07:27] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [09:07:32] Yippee, build fixed! [09:07:33] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #602: FIXED in 57 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/602/ [12:06:24] PROBLEM - Free space - all mounts on deployment-eventlogging02 is CRITICAL deployment-prep.deployment-eventlogging02.diskspace._var.byte_percentfree (<40.00%) [12:09:44] PROBLEM - App Server Main HTTP Response on deployment-mediawiki03 is CRITICAL - Socket timeout after 10 seconds [12:14:38] RECOVERY - App Server Main HTTP Response on deployment-mediawiki03 is OK: HTTP OK: HTTP/1.1 200 OK - 45556 bytes in 4.589 second response time [13:06:47] Yippee, build fixed! [13:06:48] Project browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #547: FIXED in 47 sec: https://integration.wikimedia.org/ci/job/browsertests-PageTriage-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/547/ [14:45:51] Yippee, build fixed! [14:45:51] Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #255: FIXED in 28 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/255/ [15:29:05] Yippee, build fixed! [15:29:05] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #536: FIXED in 1 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/536/ [18:00:25] PROBLEM - SSH on deployment-logstash1 is CRITICAL - Socket timeout after 10 seconds [18:01:21] 10Beta-Cluster: Make it possible to run public queries against databases on beta cluster - https://phabricator.wikimedia.org/T99456#1292144 (10Glaisher) 3NEW [18:02:40] 10Beta-Cluster, 6Labs: Make it possible to run public queries against databases on beta cluster - https://phabricator.wikimedia.org/T99456#1292152 (10Glaisher) [18:09:00] PROBLEM - App Server Main HTTP Response on deployment-mediawiki02 is CRITICAL - Socket timeout after 10 seconds [18:09:05] 6Release-Engineering, 6Project-Creators: SWAT Project (Tag) - https://phabricator.wikimedia.org/T99411#1292175 (10Krenair) looks like you already created https://phabricator.wikimedia.org/tag/2015-05-18_swat_deployment/... [18:10:14] RECOVERY - SSH on deployment-logstash1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [21:13:25] PROBLEM - SSH on deployment-logstash1 is CRITICAL - Socket timeout after 10 seconds [21:18:14] RECOVERY - SSH on deployment-logstash1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [22:32:20] PROBLEM - Puppet failure on deployment-elastic08 is CRITICAL 20.00% of data above the critical threshold [0.0] [22:32:22] PROBLEM - Puppet failure on deployment-stream is CRITICAL 20.00% of data above the critical threshold [0.0] [22:32:34] PROBLEM - Puppet failure on deployment-memc03 is CRITICAL 20.00% of data above the critical threshold [0.0] [22:33:00] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 30.00% of data above the critical threshold [0.0] [22:33:12] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 33.33% of data above the critical threshold [0.0] [22:33:26] PROBLEM - Puppet failure on deployment-cxserver03 is CRITICAL 33.33% of data above the critical threshold [0.0] [22:33:26] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 30.00% of data above the critical threshold [0.0] [22:33:26] PROBLEM - Puppet failure on deployment-bastion is CRITICAL 33.33% of data above the critical threshold [0.0] [22:33:38] PROBLEM - Puppet failure on deployment-fluoride is CRITICAL 40.00% of data above the critical threshold [0.0] [22:33:43] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 40.00% of data above the critical threshold [0.0] [22:34:07] PROBLEM - Puppet failure on deployment-fluorine is CRITICAL 33.33% of data above the critical threshold [0.0] [22:34:19] PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL 30.00% of data above the critical threshold [0.0] [22:34:41] PROBLEM - Puppet failure on deployment-test is CRITICAL 40.00% of data above the critical threshold [0.0] [22:34:41] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 40.00% of data above the critical threshold [0.0] [22:35:01] PROBLEM - Puppet failure on deployment-memc04 is CRITICAL 50.00% of data above the critical threshold [0.0] [22:35:37] PROBLEM - Puppet failure on deployment-redis01 is CRITICAL 50.00% of data above the critical threshold [0.0] [22:35:45] PROBLEM - Puppet failure on deployment-salt is CRITICAL 40.00% of data above the critical threshold [0.0] [22:36:01] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 30.00% of data above the critical threshold [0.0] [22:36:03] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL 44.44% of data above the critical threshold [0.0] [22:36:03] PROBLEM - Puppet failure on deployment-mathoid is CRITICAL 40.00% of data above the critical threshold [0.0] [22:36:07] PROBLEM - Puppet failure on deployment-upload is CRITICAL 55.56% of data above the critical threshold [0.0] [22:36:39] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 40.00% of data above the critical threshold [0.0] [22:37:05] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 50.00% of data above the critical threshold [0.0] [22:37:11] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 66.67% of data above the critical threshold [0.0] [22:37:17] PROBLEM - Puppet failure on deployment-db1 is CRITICAL 30.00% of data above the critical threshold [0.0] [22:37:55] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL 60.00% of data above the critical threshold [0.0] [22:39:58] PROBLEM - Puppet failure on deployment-test is CRITICAL 40.00% of data above the critical threshold [0.0] [22:39:58] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 40.00% of data above the critical threshold [0.0] [22:44:04] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [22:44:18] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [22:47:14] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [22:47:56] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [22:48:26] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [22:48:43] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [22:51:03] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [22:52:09] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [22:52:21] RECOVERY - Puppet failure on deployment-elastic08 is OK Less than 1.00% above the threshold [0.0] [22:53:25] RECOVERY - Puppet failure on deployment-parsoidcache02 is OK Less than 1.00% above the threshold [0.0] [22:54:41] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [22:54:41] RECOVERY - Puppet failure on deployment-test is OK Less than 1.00% above the threshold [0.0] [22:55:36] RECOVERY - Puppet failure on deployment-redis01 is OK Less than 1.00% above the threshold [0.0] [22:56:02] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [22:56:08] RECOVERY - Puppet failure on deployment-upload is OK Less than 1.00% above the threshold [0.0] [22:56:38] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [22:57:07] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [22:57:33] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [22:58:35] RECOVERY - Puppet failure on deployment-fluoride is OK Less than 1.00% above the threshold [0.0] [23:00:01] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [23:02:21] RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0] [23:02:55] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [23:03:17] RECOVERY - Puppet failure on deployment-restbase01 is OK Less than 1.00% above the threshold [0.0] [23:03:27] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [23:05:47] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [23:06:03] RECOVERY - Puppet failure on deployment-parsoid05 is OK Less than 1.00% above the threshold [0.0] [23:28:45] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<50.00%) [23:31:22] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:31:24] PROBLEM - Puppet staleness on deployment-urldownloader is CRITICAL 100.00% of data above the critical threshold [43200.0] [23:31:44] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0]