[00:04:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:07:42] PROBLEM Total processes is now: WARNING on venus.pmtpa.wmflabs 10.4.0.66 output: PROCS WARNING: 154 processes [00:12:43] RECOVERY Total processes is now: OK on venus.pmtpa.wmflabs 10.4.0.66 output: PROCS OK: 148 processes [00:13:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [00:24:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:24:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:28:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:34:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:44:03] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [00:54:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:54:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:58:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:04:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:11:53] PROBLEM Total processes is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:13:13] PROBLEM SSH is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Server answer: [01:13:43] PROBLEM Free ram is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:14:03] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [01:16:52] RECOVERY Total processes is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: PROCS OK: 102 processes [01:18:12] RECOVERY SSH is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [01:18:52] RECOVERY Free ram is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: OK: 50% free memory [01:24:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:24:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:29:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:34:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:44:03] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [01:47:42] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [01:52:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [01:54:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:55:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:59:13] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:04:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:05:53] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 9% free memory [02:10:52] PROBLEM Free ram is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Critical: 5% free memory [02:15:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [02:24:02] PROBLEM SSH is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - Socket timeout after 10 seconds [02:24:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:25:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:26:44] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 31.45, 79.30, 44.92 [02:28:53] RECOVERY SSH is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [02:29:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:34:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:35:53] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 8% free memory [02:46:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [02:51:43] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 0.08, 8.24, 19.98 [02:54:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:56:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:00:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:04:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:16:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [03:16:52] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:24:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:26:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:34:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:46:13] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [03:56:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:56:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:03:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:04:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:06:43] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 3.47, 3.82, 17.46 [04:07:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [04:12:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:16:13] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [04:17:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 252 processes [04:22:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 169 processes [04:26:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:27:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 128 processes [04:27:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:33:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:34:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:41:42] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 2.54, 2.61, 4.51 [04:46:13] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [04:56:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:58:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:03:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:04:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:16:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [05:27:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:29:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:33:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:34:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:46:53] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [05:57:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:59:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:02:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [06:03:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:04:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:07:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [06:18:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [06:27:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:29:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:30:42] PROBLEM Total processes is now: WARNING on venus.pmtpa.wmflabs 10.4.0.66 output: PROCS WARNING: 153 processes [06:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:34:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:47:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [06:48:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [06:57:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:59:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:03:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:04:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:07:43] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 2.29, 6.24, 5.62 [07:12:42] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 2.36, 3.88, 4.76 [07:18:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [07:22:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [07:25:43] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 3.66, 6.89, 5.82 [07:26:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 159 processes [07:27:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:29:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:31:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 106 processes [07:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:34:12] PROBLEM Total processes is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:34:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:34:52] PROBLEM Disk Space is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:39:13] RECOVERY Total processes is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: PROCS OK: 232 processes [07:39:53] RECOVERY Disk Space is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: DISK OK [07:48:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [07:57:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:59:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:03:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:04:54] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:07:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [08:19:02] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [08:29:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:34:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:35:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:44:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 197 processes [08:49:05] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [08:49:45] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 220 processes [08:52:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [08:54:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 186 processes [08:59:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:04:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:05:12] PROBLEM Current Load is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:06:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:10:13] RECOVERY Current Load is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK - load average: 0.16, 0.28, 0.39 [09:20:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [09:29:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:29:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:34:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:37:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:37:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [09:50:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [10:00:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:03:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:04:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:07:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:20:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [10:27:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [10:30:32] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:32:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 446 processes [10:34:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:34:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:37:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:37:43] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 3.77, 4.69, 5.36 [10:51:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [11:01:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:03:52] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 10% free memory [11:04:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:04:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:07:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:21:14] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [11:23:52] PROBLEM Free ram is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Critical: 3% free memory [11:27:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [11:28:53] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 7% free memory [11:31:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:32:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:34:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:34:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:37:08] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:42:52] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [12:01:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:04:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:04:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:07:12] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:14:53] RECOVERY dpkg-check is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: All packages OK [12:16:52] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 91% free memory [12:17:02] RECOVERY Disk Space is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: DISK OK [12:17:42] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 9.12, 7.59, 19.86 [12:17:52] RECOVERY Current Load is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK - load average: 0.04, 0.11, 0.05 [12:18:22] RECOVERY Total processes is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: PROCS OK: 90 processes [12:21:13] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [12:31:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:34:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:38:12] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:51:13] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [13:01:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:04:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:04:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:09:12] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:14:53] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 18% free memory [13:16:22] PROBLEM Total processes is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: PROCS WARNING: 162 processes [13:20:53] PROBLEM Current Load is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: WARNING - load average: 7.04, 7.59, 5.48 [13:21:13] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [13:21:23] RECOVERY Total processes is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: PROCS OK: 93 processes [13:24:53] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 86% free memory [13:25:52] RECOVERY Current Load is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK - load average: 0.05, 2.78, 3.97 [13:31:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:34:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:39:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:51:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [14:02:12] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:04:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:09:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:21:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [14:22:42] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 2.51, 3.47, 4.76 [14:24:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 319 processes [14:29:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 134 processes [14:32:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:34:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:35:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:39:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:47:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [14:51:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [14:52:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:02:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:04:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:09:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:21:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [15:32:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:34:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:35:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:39:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:51:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [16:02:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:04:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:09:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:21:53] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [16:23:53] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 9.05, 14.32, 8.27 [16:25:42] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 96.68, 46.21, 19.44 [16:30:43] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 1.03, 17.11, 14.15 [16:32:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:33:53] PROBLEM Current Load is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: CRITICAL - load average: 43.32, 26.16, 15.39 [16:34:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:35:43] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 30.75, 27.80, 18.84 [16:35:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:38:53] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 1.49, 10.42, 11.51 [16:39:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:51:53] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [17:02:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:04:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:08:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:09:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:21:53] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [17:33:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:38:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:38:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:39:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:52:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [17:59:03] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [17:59:43] PROBLEM host: bots-bnr2.pmtpa.wmflabs is DOWN address: 10.4.0.40 CRITICAL - Host Unreachable (10.4.0.40) [18:04:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:08:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:09:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:09:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:13:12] RECOVERY host: bots-bnr2.pmtpa.wmflabs is UP address: 10.4.0.40 PING OK - Packet loss = 0%, RTA = 0.76 ms [18:15:14] PROBLEM Free ram is now: UNKNOWN on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: NRPE: Unable to read output [18:17:04] RECOVERY host: bots-bnr1.pmtpa.wmflabs is UP address: 10.4.1.68 PING OK - Packet loss = 0%, RTA = 2.59 ms [18:23:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [18:35:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:38:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:39:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:39:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:53:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [19:05:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:08:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:09:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:09:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:17:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:20:32] PROBLEM dpkg-check is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [19:20:52] PROBLEM Disk Space is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:22:44] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:23:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [19:24:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 190 processes [19:25:32] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [19:25:52] RECOVERY Disk Space is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: DISK OK [19:31:52] RECOVERY dpkg-check is now: OK on follow01d.pmtpa.wmflabs 10.4.1.40 output: All packages OK [19:36:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:39:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:39:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:39:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:53:23] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [19:53:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [20:06:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:09:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:09:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [20:09:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:09:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:13:23] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [20:23:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [20:24:53] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 20.10, 11.97, 5.55 [20:26:22] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [20:29:52] RECOVERY Current Load is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: OK - load average: 1.23, 6.27, 4.95 [20:36:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:39:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:39:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:39:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:46:43] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 101.43, 66.03, 30.61 [20:53:43] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [21:01:43] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 4.57, 10.74, 15.50 [21:07:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:10:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:10:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:11:13] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:24:03] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [21:26:42] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 72.86, 25.92, 14.49 [21:37:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:41:22] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:42:12] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:43:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:44:22] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 17% free memory [21:53:52] PROBLEM Current Load is now: CRITICAL on haproxy-test2.pmtpa.wmflabs 10.4.0.252 output: Connection refused by host [21:54:32] PROBLEM Disk Space is now: CRITICAL on haproxy-test2.pmtpa.wmflabs 10.4.0.252 output: Connection refused by host [21:55:13] PROBLEM Free ram is now: CRITICAL on haproxy-test2.pmtpa.wmflabs 10.4.0.252 output: Connection refused by host [21:55:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [21:56:43] PROBLEM Total processes is now: CRITICAL on haproxy-test2.pmtpa.wmflabs 10.4.0.252 output: Connection refused by host [21:57:23] PROBLEM dpkg-check is now: CRITICAL on haproxy-test2.pmtpa.wmflabs 10.4.0.252 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:07:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:12:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:13:14] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:13:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:25:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [22:37:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:42:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:43:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:43:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:55:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [23:07:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [23:12:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:13:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:13:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:26:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [23:37:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [23:42:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:43:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:43:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:56:14] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93)