[00:23:55] PROBLEM Current Load is now: CRITICAL on mwreview-dev9.pmtpa.wmflabs 10.4.1.103 output: Connection refused by host [00:24:35] PROBLEM Disk Space is now: CRITICAL on mwreview-dev9.pmtpa.wmflabs 10.4.1.103 output: Connection refused by host [00:25:15] PROBLEM Free ram is now: CRITICAL on mwreview-dev9.pmtpa.wmflabs 10.4.1.103 output: Connection refused by host [00:26:45] PROBLEM Total processes is now: CRITICAL on mwreview-dev9.pmtpa.wmflabs 10.4.1.103 output: Connection refused by host [00:27:25] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev9.pmtpa.wmflabs 10.4.1.103 output: Connection refused by host [00:38:45] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 269 processes [01:00:15] PROBLEM SSH is now: CRITICAL on accounts-database.pmtpa.wmflabs 10.4.1.33 output: Server answer: [01:07:46] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 330 processes [01:13:24] PROBLEM Free ram is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: Critical: 2% free memory [01:17:44] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 99 processes [01:18:24] RECOVERY Free ram is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: OK: 91% free memory [01:35:15] RECOVERY SSH is now: OK on accounts-database.pmtpa.wmflabs 10.4.1.33 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [01:43:45] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 200 processes [02:08:46] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 227 processes [03:01:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 194 processes [03:06:26] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 103 processes [03:16:45] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 194 processes [03:21:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 430 processes [03:35:26] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 170 processes [04:16:40] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 5% free memory [04:18:58] PROBLEM Current Load is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: Connection refused by host [04:18:58] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [04:19:38] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: Connection refused by host [04:20:17] PROBLEM Free ram is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: Connection refused by host [04:21:47] PROBLEM Total processes is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: Connection refused by host [04:22:27] PROBLEM dpkg-check is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: Connection refused by host [06:29:45] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS CRITICAL: 202 processes [06:39:43] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 197 processes [06:49:42] PROBLEM Total processes is now: WARNING on bots-ibnr1.pmtpa.wmflabs 10.4.0.232 output: PROCS WARNING: 152 processes [06:54:43] RECOVERY Total processes is now: OK on bots-ibnr1.pmtpa.wmflabs 10.4.0.232 output: PROCS OK: 143 processes [07:48:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 174 processes [07:58:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 134 processes [09:31:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 158 processes [09:41:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 144 processes [10:06:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 170 processes [11:47:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [12:17:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 166 processes [12:32:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 135 processes [12:42:32] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 6.01, 6.05, 5.26 [12:57:32] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.46, 4.03, 4.76 [13:45:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 168 processes [13:46:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 194 processes [13:48:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 178 processes [13:53:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 207 processes [13:56:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 214 processes [14:33:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [14:39:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 191 processes [14:46:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 192 processes [15:09:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 206 processes [15:39:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [16:03:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 207 processes [16:08:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 225 processes [16:13:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 197 processes [16:18:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 141 processes [16:54:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 236 processes [17:26:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 265 processes [17:56:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 193 processes [18:01:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [18:46:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [19:02:42] PROBLEM Disk Space is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: DISK WARNING - free space: /logs_db 600 MB (5% inode=-): [19:10:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 29.26, 18.78, 9.84 [19:14:43] PROBLEM Total processes is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS CRITICAL: 202 processes [19:17:33] RECOVERY Disk Space is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: DISK OK [19:19:43] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 198 processes [19:21:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 384 processes [20:03:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 171 processes [20:10:33] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 34.04, 29.42, 23.17 [20:13:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 210 processes [20:20:35] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 10.74, 16.69, 19.98 [20:31:03] RECOVERY Current Load is now: OK on zero-vmod.pmtpa.wmflabs 10.4.1.101 output: OK - load average: 1.21, 0.33, 0.11 [20:31:53] RECOVERY Total processes is now: OK on zero-vmod.pmtpa.wmflabs 10.4.1.101 output: PROCS OK: 93 processes [20:32:33] RECOVERY dpkg-check is now: OK on zero-vmod.pmtpa.wmflabs 10.4.1.101 output: All packages OK [20:34:43] RECOVERY Disk Space is now: OK on zero-vmod.pmtpa.wmflabs 10.4.1.101 output: DISK OK [20:35:22] PROBLEM Free ram is now: UNKNOWN on zero-vmod.pmtpa.wmflabs 10.4.1.101 output: NRPE: Unable to read output [20:59:43] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:02:53] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 32% free memory [21:04:43] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 198 processes [21:43:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 192 processes [21:58:53] PROBLEM Current Load is now: CRITICAL on rt-testing5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [21:59:33] PROBLEM Disk Space is now: CRITICAL on rt-testing5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [22:00:13] PROBLEM Free ram is now: CRITICAL on rt-testing5.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [22:01:43] PROBLEM Total processes is now: CRITICAL on rt-testing5.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:02:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing5.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:03:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 201 processes [22:25:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 14.56, 12.15, 14.71 [22:53:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 168 processes [23:23:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 521 processes [23:53:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 197 processes