[00:16:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 196 processes [00:46:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 292 processes [01:13:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 310 processes [01:20:33] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 23.77, 23.26, 20.41 [01:28:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 106 processes [01:46:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 237 processes [02:30:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 11.75, 17.06, 19.19 [03:10:33] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 23.05, 21.47, 20.01 [03:48:53] PROBLEM Current Load is now: CRITICAL on rt-testing9.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [03:49:33] PROBLEM Disk Space is now: CRITICAL on rt-testing9.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [03:50:13] PROBLEM Free ram is now: CRITICAL on rt-testing9.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [03:51:43] PROBLEM Total processes is now: CRITICAL on rt-testing9.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [03:52:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing9.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [05:19:02] PROBLEM Current Load is now: CRITICAL on testing-newimage4.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Socket timeout after 10 seconds. [05:19:32] PROBLEM Disk Space is now: CRITICAL on testing-newimage4.pmtpa.wmflabs 10.4.0.142 output: Connection refused or timed out [05:24:03] PROBLEM host: testing-newimage4.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:29:44] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS CRITICAL: 204 processes [06:54:43] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 199 processes [07:33:53] PROBLEM Current Load is now: CRITICAL on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:34:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:35:19] PROBLEM Free ram is now: CRITICAL on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:36:43] PROBLEM Total processes is now: CRITICAL on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: Connection refused or timed out [07:37:33] PROBLEM dpkg-check is now: CRITICAL on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: Connection refused or timed out [07:41:43] RECOVERY Total processes is now: OK on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: PROCS OK: 82 processes [07:42:23] RECOVERY dpkg-check is now: OK on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: All packages OK [07:43:53] RECOVERY Current Load is now: OK on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: OK - load average: 0.08, 0.17, 0.08 [07:44:33] RECOVERY Disk Space is now: OK on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: DISK OK [07:45:22] PROBLEM Free ram is now: UNKNOWN on testing-newimage5.pmtpa.wmflabs 10.4.0.142 output: NRPE: Unable to read output [08:03:52] PROBLEM Current Load is now: CRITICAL on testing-oldimage5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [08:04:32] PROBLEM Disk Space is now: CRITICAL on testing-oldimage5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [08:05:12] PROBLEM Free ram is now: CRITICAL on testing-oldimage5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [08:06:42] PROBLEM Total processes is now: CRITICAL on testing-oldimage5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [08:07:22] PROBLEM dpkg-check is now: CRITICAL on testing-oldimage5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [08:38:52] PROBLEM Current Load is now: CRITICAL on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:39:32] PROBLEM Disk Space is now: CRITICAL on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:40:13] PROBLEM Free ram is now: CRITICAL on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:41:53] PROBLEM Total processes is now: CRITICAL on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Socket timeout after 10 seconds. [08:43:53] RECOVERY Current Load is now: OK on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: OK - load average: 0.92, 0.58, 0.23 [08:44:33] RECOVERY Disk Space is now: OK on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: DISK OK [08:45:12] PROBLEM Free ram is now: UNKNOWN on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: NRPE: Unable to read output [08:46:42] RECOVERY Total processes is now: OK on testing-newimage6.pmtpa.wmflabs 10.4.1.102 output: PROCS OK: 80 processes [08:54:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage7.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [08:55:13] PROBLEM Free ram is now: CRITICAL on testing-newimage7.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:55:53] PROBLEM Current Load is now: CRITICAL on testing-newimage7.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:56:43] PROBLEM Total processes is now: CRITICAL on testing-newimage7.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:57:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage7.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:07:12] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 5% free memory [09:22:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 165 processes [09:24:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage8.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [09:25:13] PROBLEM Free ram is now: CRITICAL on testing-newimage8.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:26:03] PROBLEM Current Load is now: CRITICAL on testing-newimage8.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Socket timeout after 10 seconds. [09:26:43] PROBLEM Total processes is now: CRITICAL on testing-newimage8.pmtpa.wmflabs 10.4.0.142 output: Connection refused or timed out [09:27:33] PROBLEM dpkg-check is now: CRITICAL on testing-newimage8.pmtpa.wmflabs 10.4.0.142 output: Connection refused or timed out [09:32:32] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 143 processes [09:33:52] PROBLEM Current Load is now: CRITICAL on testing-newimage9.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:34:32] PROBLEM Disk Space is now: CRITICAL on testing-newimage9.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:35:13] PROBLEM Free ram is now: CRITICAL on testing-newimage9.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:36:43] PROBLEM Total processes is now: CRITICAL on testing-newimage9.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:37:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage9.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:38:53] PROBLEM Current Load is now: CRITICAL on testing-newimage10.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [09:39:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage10.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [09:40:13] PROBLEM Free ram is now: CRITICAL on testing-newimage10.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [09:41:43] PROBLEM Total processes is now: CRITICAL on testing-newimage10.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [09:42:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage10.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [09:58:53] PROBLEM Current Load is now: CRITICAL on testing-newimage11.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [09:59:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage11.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [10:00:13] PROBLEM Free ram is now: CRITICAL on testing-newimage11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:01:43] PROBLEM Total processes is now: CRITICAL on testing-newimage11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:02:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage11.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:03:53] PROBLEM Current Load is now: CRITICAL on testing-oldimage6.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [10:04:33] PROBLEM Disk Space is now: CRITICAL on testing-oldimage6.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [10:05:12] PROBLEM Free ram is now: CRITICAL on testing-oldimage6.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [10:06:42] PROBLEM Total processes is now: CRITICAL on testing-oldimage6.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [10:07:22] PROBLEM dpkg-check is now: CRITICAL on testing-oldimage6.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [10:18:53] PROBLEM Current Load is now: CRITICAL on testing-newimage12.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [10:19:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage12.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:20:12] PROBLEM Free ram is now: CRITICAL on testing-newimage12.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:21:42] PROBLEM Total processes is now: CRITICAL on testing-newimage12.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:22:22] PROBLEM dpkg-check is now: CRITICAL on testing-newimage12.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:25:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 186 processes [11:04:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 189 processes [11:09:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 237 processes [12:09:53] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [13:09:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 169 processes [13:14:32] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 117 processes [13:27:33] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 201 processes [14:17:12] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [14:27:13] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [15:22:13] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [15:30:52] PROBLEM dpkg-check is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [15:31:12] PROBLEM Total processes is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [15:33:42] PROBLEM Disk Space is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [15:33:42] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:35:53] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:36:13] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:38:43] RECOVERY Disk Space is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: DISK OK [15:38:43] RECOVERY Current Load is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: OK - load average: 0.41, 0.89, 1.05 [15:40:53] RECOVERY dpkg-check is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: All packages OK [15:41:13] RECOVERY Total processes is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: PROCS OK: 100 processes [15:52:13] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [16:09:43] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:10:53] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [16:15:52] RECOVERY SSH is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [16:34:42] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:47:43] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:48:53] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [17:52:13] PROBLEM Free ram is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: Warning: 19% free memory [17:53:53] RECOVERY SSH is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [18:04:53] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:06:53] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [18:07:43] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:08:13] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:27:53] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:30:43] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:31:13] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:42:13] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [19:47:13] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [21:09:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory