[00:08:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [00:09:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:12:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:13:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 211 processes [00:14:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:17:45] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [00:25:34] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [00:27:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [00:38:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [00:39:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:43:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:44:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:45:33] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 5% free memory [00:47:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 193 processes [00:50:33] PROBLEM Current Load is now: WARNING on bots-apache01.pmtpa.wmflabs 10.4.0.141 output: WARNING - load average: 7.35, 13.83, 6.59 [00:55:35] RECOVERY Current Load is now: OK on bots-apache01.pmtpa.wmflabs 10.4.0.141 output: OK - load average: 0.05, 5.07, 4.77 [00:55:35] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [00:57:45] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [01:10:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:10:53] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [01:12:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 479 processes [01:12:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [01:13:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:14:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:17:45] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 105 processes [01:17:55] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 26% free memory [01:25:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [01:30:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [01:40:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:40:53] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [01:44:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:44:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:55:55] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [02:01:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [02:07:54] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 2.49, 2.68, 4.90 [02:10:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:10:53] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [02:14:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:14:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:17:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 187 processes [02:25:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.80, 9.38, 6.93 [02:27:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [02:31:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [02:32:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 214 processes [02:41:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [02:41:48] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:42:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [02:44:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:45:25] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:57:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [03:01:24] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [03:11:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:12:33] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [03:14:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:16:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:24:24] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 313 processes [03:27:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [03:31:25] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [03:41:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:42:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [03:44:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:47:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:57:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [04:02:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [04:02:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [04:13:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [04:13:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:16:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:18:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:22:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 211 processes [04:27:15] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [04:32:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [04:43:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [04:43:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:46:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:48:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:51:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 0.83, 1.42, 4.21 [04:55:25] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 3.11, 3.24, 4.67 [04:57:15] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [05:02:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [05:03:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.23, 1.87, 4.74 [05:04:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 0.52, 0.96, 4.45 [05:05:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.36, 1.68, 4.44 [05:07:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 1.37, 1.43, 4.07 [05:13:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [05:13:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:16:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:18:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:27:44] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [05:32:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [05:43:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [05:43:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:46:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:48:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:57:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [06:02:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [06:13:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [06:13:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:16:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:18:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:19:04] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [06:27:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [06:32:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [06:35:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 13.09, 12.01, 7.69 [06:36:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.83, 8.92, 6.03 [06:37:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 12.40, 10.87, 7.01 [06:38:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 11.63, 10.05, 6.23 [06:43:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [06:44:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:44:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 11.20, 9.65, 6.86 [06:47:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:48:25] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 10.79, 10.06, 7.28 [06:48:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:57:53] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [07:02:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [07:13:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [07:14:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:17:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:18:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:27:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [07:32:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [07:43:33] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [07:44:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:47:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:49:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:57:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [08:02:43] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [08:15:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:15:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [08:17:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:19:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:20:44] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [08:25:45] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:27:55] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [08:30:46] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [08:32:43] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [08:45:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:45:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [08:47:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:49:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:00:32] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [09:02:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [09:06:55] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.13, 1.51, 4.10 [09:07:55] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 1.20, 1.51, 4.54 [09:08:45] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.09, 3.15, 5.00 [09:10:54] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 1.48, 1.48, 4.18 [09:15:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:15:55] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [09:17:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:19:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:30:33] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [09:33:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [09:46:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:47:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [09:48:24] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 1.02, 1.48, 4.65 [09:49:44] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 1.06, 1.29, 4.25 [09:49:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:52:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 187 processes [09:54:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:00:34] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [10:04:24] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [10:18:23] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [10:18:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:19:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:24:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:30:53] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [10:35:26] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [10:48:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [10:49:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:51:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:54:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:00:53] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [11:05:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [11:18:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [11:19:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:22:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:24:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:32:12] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [11:35:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [11:48:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [11:49:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:52:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:54:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:02:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [12:05:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [12:17:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 211 processes [12:18:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [12:19:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:22:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:24:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:32:43] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [12:35:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [12:48:33] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [12:49:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:53:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:55:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:02:43] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [13:05:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [13:18:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [13:19:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:23:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:26:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:33:02] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [13:36:03] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [13:48:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [13:49:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:54:04] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:57:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:00:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [14:05:35] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [14:05:45] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [14:06:15] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [14:18:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [14:19:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:24:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:28:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:35:35] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [14:36:15] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [14:45:32] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [14:50:02] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [14:50:02] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:50:33] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 6% free memory [14:54:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:58:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:05:52] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [15:06:22] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [15:15:34] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [15:20:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:20:53] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [15:24:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:28:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:35:59] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [15:36:24] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [15:51:50] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:52:02] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [15:54:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:58:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:02:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 193 processes [16:05:33] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [16:06:03] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [16:07:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [16:15:42] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [16:21:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:22:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [16:22:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [16:24:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:27:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 187 processes [16:28:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:34:44] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 193 processes [16:37:04] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [16:37:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [16:37:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [16:39:44] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 262 processes [16:50:33] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [16:51:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:52:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [16:54:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:58:54] PROBLEM Current Load is now: CRITICAL on rt-testing2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [16:58:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:59:34] PROBLEM Disk Space is now: CRITICAL on rt-testing2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [17:00:22] PROBLEM Free ram is now: CRITICAL on rt-testing2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [17:01:42] PROBLEM Total processes is now: CRITICAL on rt-testing2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [17:02:22] PROBLEM dpkg-check is now: CRITICAL on rt-testing2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [17:07:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [17:07:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [17:18:54] PROBLEM Current Load is now: CRITICAL on rt-testing3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [17:19:34] PROBLEM Disk Space is now: CRITICAL on rt-testing3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [17:20:13] PROBLEM Free ram is now: CRITICAL on rt-testing3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [17:21:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:21:43] PROBLEM Total processes is now: CRITICAL on rt-testing3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [17:22:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [17:22:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [17:24:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:25:35] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [17:28:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:37:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [17:37:35] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [17:51:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:52:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [17:54:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:59:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:00:33] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [18:07:15] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [18:07:35] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [18:14:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.65, 7.68, 5.94 [18:16:23] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 8.11, 7.56, 5.84 [18:21:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:22:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 7.21, 6.90, 5.70 [18:22:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [18:23:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 9.88, 9.35, 6.75 [18:25:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 7.47, 6.78, 5.53 [18:26:05] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:29:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:36:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.28, 9.27, 6.54 [18:37:44] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [18:37:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [18:52:53] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [18:52:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:56:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:02:32] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:07:44] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [19:07:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [19:23:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [19:23:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:26:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:26:44] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 153 processes [19:33:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:37:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [19:37:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [19:39:24] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 237 processes [19:53:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [19:53:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:55:44] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [19:56:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:00:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [20:04:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:07:53] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [20:08:43] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [20:23:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [20:23:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:27:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:34:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:37:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [20:39:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [20:53:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [20:53:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:57:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:04:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:07:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [21:10:53] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [21:21:43] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS CRITICAL: 204 processes [21:23:33] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [21:23:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:29:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:34:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:37:58] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [21:41:43] PROBLEM Disk Space is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK WARNING - free space: /tmp 0 MB (5% inode=99%): [21:43:33] PROBLEM Lucene frontend is now: CRITICAL on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: CRITICAL - Socket timeout after 10 seconds [22:19:25] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [22:21:45] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK CRITICAL - free space: /tmp 0 MB (1% inode=99%): [22:24:25] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Unable to read output [22:39:44] PROBLEM Lucene frontend is now: CRITICAL on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: CRITICAL - Socket timeout after 10 seconds [22:47:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 187 processes [22:57:14] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [22:57:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 180 processes [23:02:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [23:02:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [23:07:25] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 106 processes [23:34:18] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [23:42:12] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:43:42] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:43:42] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:47:55] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:52:12] PROBLEM Total processes is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [23:53:42] PROBLEM Disk Space is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [23:57:14] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:57:54] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [23:58:34] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake.