[00:04:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:04:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:13:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:19:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:34:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:35:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:43:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:48:42] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [00:49:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:58:32] RECOVERY SSH is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [01:04:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:06:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:09:52] PROBLEM Free ram is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:11:23] PROBLEM Current Load is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:12:13] PROBLEM Disk Space is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:12:53] PROBLEM Total processes is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:13:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:13:43] PROBLEM dpkg-check is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:14:43] RECOVERY Free ram is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: OK: 55% free memory [01:16:22] RECOVERY Current Load is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: OK - load average: 0.88, 4.61, 3.20 [01:17:22] RECOVERY Disk Space is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: DISK OK [01:17:52] RECOVERY Total processes is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: PROCS OK: 99 processes [01:18:42] RECOVERY dpkg-check is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: All packages OK [01:19:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:34:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:36:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:43:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:49:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:03:42] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [02:05:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:06:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:13:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:19:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:24:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory [02:36:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:36:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:39:43] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 20% free memory [02:43:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:47:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory [02:49:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:06:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:07:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:13:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:22:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:37:13] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:38:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:43:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:52:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:07:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:08:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:14:02] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:22:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:37:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:38:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:44:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:52:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:07:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:08:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:14:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:22:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:37:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:38:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:46:01] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:52:12] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:07:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:08:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:15:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:23:12] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:35:42] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [06:36:52] RECOVERY Disk Space is now: OK on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: DISK OK [06:39:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:39:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:45:32] RECOVERY SSH is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [06:46:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:54:12] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:09:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:09:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:16:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:23:43] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [07:24:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:39:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:39:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:46:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:54:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:09:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:10:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:18:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:24:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:33:42] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [08:39:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:41:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:48:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:54:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:09:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:11:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:18:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:24:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:40:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:41:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:48:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:54:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:11:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:12:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:18:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:24:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:34:32] PROBLEM Disk Space is now: CRITICAL on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [10:34:32] PROBLEM Free ram is now: CRITICAL on maps-tiles2.pmtpa.wmflabs 10.4.1.0 output: Connection refused by host [10:35:12] PROBLEM Free ram is now: CRITICAL on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [10:35:12] PROBLEM Total processes is now: CRITICAL on maps-varnish.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [10:35:52] PROBLEM Current Load is now: CRITICAL on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [10:35:52] PROBLEM Disk Space is now: CRITICAL on maps-tiles2.pmtpa.wmflabs 10.4.1.0 output: Connection refused by host [10:35:52] PROBLEM Free ram is now: CRITICAL on maps-varnish.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [10:36:02] PROBLEM Total processes is now: CRITICAL on maps-tiles2.pmtpa.wmflabs 10.4.1.0 output: Connection refused by host [10:36:02] PROBLEM dpkg-check is now: CRITICAL on maps-varnish.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [10:36:42] PROBLEM Total processes is now: CRITICAL on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [10:36:42] PROBLEM dpkg-check is now: CRITICAL on maps-tiles2.pmtpa.wmflabs 10.4.1.0 output: Connection refused by host [10:37:22] PROBLEM dpkg-check is now: CRITICAL on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: Connection refused by host [10:37:22] PROBLEM Current Load is now: CRITICAL on maps-varnish.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [10:37:52] PROBLEM Current Load is now: CRITICAL on maps-tiles2.pmtpa.wmflabs 10.4.1.0 output: Connection refused by host [10:38:12] PROBLEM Disk Space is now: CRITICAL on maps-varnish.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [10:39:32] PROBLEM Current Load is now: CRITICAL on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: Connection refused by host [10:40:13] PROBLEM Disk Space is now: CRITICAL on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: Connection refused by host [10:41:03] PROBLEM Free ram is now: CRITICAL on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: Connection refused by host [10:42:03] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:42:24] PROBLEM Total processes is now: CRITICAL on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: Connection refused by host [10:42:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:42:53] PROBLEM dpkg-check is now: CRITICAL on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: Connection refused by host [10:48:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:54:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:12:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:13:03] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:19:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:23:43] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [11:24:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:43:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:44:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:49:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:54:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:02:42] RECOVERY dpkg-check is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: All packages OK [12:03:22] RECOVERY Current Load is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: OK - load average: 0.51, 0.32, 0.13 [12:03:46] RECOVERY Free ram is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: OK: 87% free memory [12:03:52] RECOVERY Disk Space is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: DISK OK [12:04:02] RECOVERY Total processes is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: PROCS OK: 90 processes [12:08:52] PROBLEM Current Load is now: CRITICAL on maps-static.pmtpa.wmflabs 10.4.0.213 output: Connection refused by host [12:09:32] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [12:09:32] PROBLEM Disk Space is now: CRITICAL on maps-static.pmtpa.wmflabs 10.4.0.213 output: Connection refused by host [12:10:14] PROBLEM Free ram is now: CRITICAL on maps-static.pmtpa.wmflabs 10.4.0.213 output: Connection refused by host [12:10:54] PROBLEM SSH is now: CRITICAL on accounts-application.pmtpa.wmflabs 10.4.0.147 output: Connection refused [12:11:44] PROBLEM Total processes is now: CRITICAL on maps-static.pmtpa.wmflabs 10.4.0.213 output: Connection refused by host [12:12:24] PROBLEM dpkg-check is now: CRITICAL on maps-static.pmtpa.wmflabs 10.4.0.213 output: Connection refused by host [12:13:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:14:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:16:02] RECOVERY SSH is now: OK on accounts-application.pmtpa.wmflabs 10.4.0.147 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [12:20:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:23:53] PROBLEM SSH is now: CRITICAL on accounts-application.pmtpa.wmflabs 10.4.0.147 output: Connection refused [12:24:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:40:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [12:44:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:44:33] RECOVERY Current Load is now: OK on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: OK - load average: 0.32, 0.13, 0.05 [12:45:14] RECOVERY Disk Space is now: OK on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: DISK OK [12:45:24] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:51:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:52:32] PROBLEM Current Load is now: CRITICAL on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: Connection refused by host [12:53:12] PROBLEM Disk Space is now: CRITICAL on maps-wikiminiatlas.pmtpa.wmflabs 10.4.0.234 output: Connection refused by host [12:56:13] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:10:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [13:14:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:16:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:21:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:27:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:41:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [13:44:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:47:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:53:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:57:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:11:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [14:14:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:17:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:23:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:29:12] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:41:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [14:44:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:47:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:53:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:59:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:11:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [15:14:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:17:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:23:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:29:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:41:13] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [15:44:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:44:44] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [15:48:32] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:53:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:59:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:11:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [16:15:52] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [16:18:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:19:32] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:24:02] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:29:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:37:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:41:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [16:42:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [16:45:53] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [16:48:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:49:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:54:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:59:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:11:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [17:16:32] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [17:19:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:20:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:24:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:29:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:41:53] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [17:42:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [17:46:42] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [17:49:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:50:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:54:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:59:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:07:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [18:11:53] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [18:17:32] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [18:19:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:21:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:24:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:29:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:42:21] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [18:47:42] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [18:49:41] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:51:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:52:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [18:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:59:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:13:23] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [19:17:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:17:43] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [19:19:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:21:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:27:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [19:29:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:43:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [19:47:42] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [19:49:32] PROBLEM host: openid-wiki.pmtpa.wmflabs is DOWN address: 10.4.1.50 CRITICAL - Host Unreachable (10.4.1.50) [19:49:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:51:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:59:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:13:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [20:17:42] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [20:19:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:19:42] PROBLEM host: openid-wiki.pmtpa.wmflabs is DOWN address: 10.4.1.50 CRITICAL - Host Unreachable (10.4.1.50) [20:21:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:29:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:42:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [20:43:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [20:47:43] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [20:49:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:49:43] PROBLEM host: openid-wiki.pmtpa.wmflabs is DOWN address: 10.4.1.50 CRITICAL - Host Unreachable (10.4.1.50) [20:52:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:55:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:59:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:04:42] RECOVERY Current Load is now: OK on openid-wiki.pmtpa.wmflabs 10.4.1.50 output: OK - load average: 0.42, 0.16, 0.06 [21:04:52] RECOVERY host: openid-wiki.pmtpa.wmflabs is UP address: 10.4.1.50 PING OK - Packet loss = 0%, RTA = 0.86 ms [21:05:42] RECOVERY Disk Space is now: OK on openid-wiki.pmtpa.wmflabs 10.4.1.50 output: DISK OK [21:07:22] RECOVERY Free ram is now: OK on openid-wiki.pmtpa.wmflabs 10.4.1.50 output: OK: 88% free memory [21:07:32] RECOVERY dpkg-check is now: OK on openid-wiki.pmtpa.wmflabs 10.4.1.50 output: All packages OK [21:08:42] RECOVERY Total processes is now: OK on openid-wiki.pmtpa.wmflabs 10.4.1.50 output: PROCS OK: 91 processes [21:13:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [21:17:52] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [21:22:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:23:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:28:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:32:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:43:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [21:47:52] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [21:52:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:53:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:58:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:04:22] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:13:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [22:15:53] PROBLEM SSH is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Server answer: [22:17:53] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [22:24:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:24:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:28:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:30:52] RECOVERY SSH is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:34:22] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:43:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [22:47:53] PROBLEM host: bots-bnr1.pmtpa.wmflabs is DOWN address: 10.4.1.68 CRITICAL - Host Unreachable (10.4.1.68) [22:54:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:54:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:56:03] RECOVERY host: bots-bnr1.pmtpa.wmflabs is UP address: 10.4.1.68 PING OK - Packet loss = 0%, RTA = 0.86 ms [22:56:33] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 130 processes [22:57:23] RECOVERY Disk Space is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: DISK OK [22:57:43] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 0.26, 0.45, 0.21 [22:57:53] RECOVERY Free ram is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK: 95% free memory [22:58:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:58:53] RECOVERY dpkg-check is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: All packages OK [23:04:22] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:04:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 758 processes [23:13:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [23:24:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:24:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [23:28:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:32:32] RECOVERY Disk Space is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: DISK OK [23:32:52] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 6.12, 2.58, 0.98 [23:33:42] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 168 processes [23:34:12] RECOVERY Free ram is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK: 89% free memory [23:34:22] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:34:22] RECOVERY dpkg-check is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: All packages OK [23:43:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [23:45:42] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 4.96, 9.03, 5.64 [23:45:52] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 5.87, 8.22, 5.69 [23:47:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [23:52:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [23:54:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:54:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [23:55:42] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 7.85, 4.87, 4.61 [23:55:52] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.21, 3.81, 4.49 [23:58:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122)