[00:17:23] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 28% free memory [00:44:53] PROBLEM Free ram is now: UNKNOWN on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Unknown [00:49:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 1% free memory [00:53:43] PROBLEM SSH is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Server answer: [00:54:33] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:54:43] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:54:53] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:54:53] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:58:43] RECOVERY SSH is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:59:33] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.35, 1.45, 1.12 [00:59:43] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 200 processes [00:59:53] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 33% free memory [00:59:53] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [00:59:53] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [01:09:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 237 processes [01:14:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [01:19:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes [01:19:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [02:43:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 183 processes [02:53:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 100 processes [03:29:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [03:34:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:14:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [04:21:42] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 178 processes [04:26:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 110 processes [04:49:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [05:34:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [06:29:45] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS CRITICAL: 205 processes [06:47:02] PROBLEM Total processes is now: WARNING on accounts-database.pmtpa.wmflabs 10.4.1.33 output: PROCS WARNING: 155 processes [06:59:43] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 200 processes [07:19:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [07:24:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [10:49:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:34:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [12:09:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [12:29:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [12:59:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:15:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [13:54:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 189 processes [13:59:42] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 106 processes [14:12:32] PROBLEM dpkg-check is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: DPKG CRITICAL dpkg reports broken packages [14:17:33] RECOVERY dpkg-check is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: All packages OK [14:50:43] PROBLEM dpkg-check is now: CRITICAL on tools-shadow.pmtpa.wmflabs 10.4.1.66 output: DPKG CRITICAL dpkg reports broken packages [14:55:42] RECOVERY dpkg-check is now: OK on tools-shadow.pmtpa.wmflabs 10.4.1.66 output: All packages OK [15:18:52] PROBLEM Current Load is now: CRITICAL on rt-testing10.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:19:32] PROBLEM Disk Space is now: CRITICAL on rt-testing10.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:20:13] PROBLEM Free ram is now: CRITICAL on rt-testing10.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:21:43] PROBLEM Total processes is now: CRITICAL on rt-testing10.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:22:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing10.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:07:42] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 248 processes [16:17:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 115 processes [16:23:52] PROBLEM Current Load is now: CRITICAL on rt-testing11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:24:32] PROBLEM Disk Space is now: CRITICAL on rt-testing11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:25:13] PROBLEM Free ram is now: CRITICAL on rt-testing11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:26:43] PROBLEM Total processes is now: CRITICAL on rt-testing11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:27:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:00:53] PROBLEM Current Load is now: CRITICAL on rt-testing12.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [17:01:33] PROBLEM Disk Space is now: CRITICAL on rt-testing12.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [17:01:43] PROBLEM Total processes is now: CRITICAL on rt-testing12.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [17:02:13] PROBLEM Free ram is now: CRITICAL on rt-testing12.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:02:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing12.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:51:43] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 254 processes [18:01:49] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 184 processes [18:04:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [18:06:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 113 processes [18:09:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:04:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [19:34:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [20:04:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [20:18:53] PROBLEM Current Load is now: CRITICAL on testing-newimage24.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [20:19:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage24.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:20:13] PROBLEM Free ram is now: CRITICAL on testing-newimage24.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:21:43] PROBLEM Total processes is now: CRITICAL on testing-newimage24.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:22:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage24.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:34:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [20:38:03] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [20:57:23] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 152 processes [21:58:22] PROBLEM SSH is now: CRITICAL on accounts-database.pmtpa.wmflabs 10.4.1.33 output: Server answer: [22:08:23] RECOVERY SSH is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:12:02] RECOVERY Total processes is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: PROCS OK: 87 processes [22:25:12] PROBLEM Free ram is now: CRITICAL on testing-newimage25.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [22:25:52] PROBLEM Current Load is now: CRITICAL on testing-newimage25.pmtpa.wmflabs 10.4.0.27 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:26:32] PROBLEM Disk Space is now: CRITICAL on testing-newimage25.pmtpa.wmflabs 10.4.0.27 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:26:42] PROBLEM Total processes is now: CRITICAL on testing-newimage25.pmtpa.wmflabs 10.4.0.27 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:27:22] PROBLEM dpkg-check is now: CRITICAL on testing-newimage25.pmtpa.wmflabs 10.4.0.27 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:34:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage26.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [22:35:13] PROBLEM Free ram is now: CRITICAL on testing-newimage26.pmtpa.wmflabs 10.4.0.119 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:35:57] PROBLEM Current Load is now: CRITICAL on testing-newimage26.pmtpa.wmflabs 10.4.0.119 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:36:43] PROBLEM Total processes is now: CRITICAL on testing-newimage26.pmtpa.wmflabs 10.4.0.119 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:37:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage26.pmtpa.wmflabs 10.4.0.119 output: CHECK_NRPE: Error - Could not complete SSL handshake.