[00:16:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 399 processes [00:45:53] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 128 processes [00:46:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 144 processes [00:47:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 158 processes [00:52:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 335 processes [00:53:53] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 311 processes [00:54:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 330 processes [01:13:32] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 151 processes [01:45:53] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 158 processes [01:48:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 156 processes [01:53:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 128 processes [01:56:43] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 2.82, 3.60, 4.95 [02:03:52] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK CRITICAL - free space: / 8 MB (0% inode=72%): [02:23:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 194 processes [02:28:53] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 206 processes [02:53:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 195 processes [03:03:33] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 150 processes [03:11:32] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 152 processes [03:14:12] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [03:17:55] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 194 processes [03:19:15] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [03:21:33] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 150 processes [03:44:13] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [03:48:02] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 209 processes [03:57:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 194 processes [04:13:58] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 230 processes [04:14:38] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 152 processes [04:20:53] PROBLEM Total processes is now: CRITICAL on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS CRITICAL: 213 processes [04:25:58] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 200 processes [04:45:52] PROBLEM Total processes is now: CRITICAL on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS CRITICAL: 202 processes [05:30:53] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 155 processes [05:32:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 281 processes [06:04:16] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [06:10:55] RECOVERY Total processes is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS OK: 148 processes [06:28:55] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 171 processes [07:13:02] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 188 processes [07:14:22] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [07:19:22] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [07:23:02] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 224 processes [09:14:21] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [09:19:23] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [09:28:51] PROBLEM Total processes is now: CRITICAL on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS CRITICAL: 201 processes [09:38:53] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 195 processes [09:44:22] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [09:48:03] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 176 processes [09:48:54] RECOVERY Total processes is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS OK: 145 processes [09:49:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 198 processes [09:54:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 189 processes [09:59:42] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 150 processes [09:59:52] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 147 processes [10:29:22] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [10:33:03] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 146 processes [10:37:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 156 processes [11:32:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 222 processes [11:33:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 210 processes [11:35:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 194 processes [11:50:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 290 processes [12:15:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 167 processes [12:18:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 174 processes [12:23:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 202 processes [12:32:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 156 processes [13:32:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 207 processes [13:34:33] RECOVERY Disk Space is now: OK on kraken0.pmtpa.wmflabs 10.4.0.142 output: DISK OK [13:35:53] RECOVERY Current Load is now: OK on kraken0.pmtpa.wmflabs 10.4.0.142 output: OK - load average: 0.05, 0.10, 0.05 [13:36:03] PROBLEM Free ram is now: UNKNOWN on kraken0.pmtpa.wmflabs 10.4.0.142 output: NRPE: Unable to read output [13:36:43] RECOVERY Total processes is now: OK on kraken0.pmtpa.wmflabs 10.4.0.142 output: PROCS OK: 87 processes [13:37:33] RECOVERY dpkg-check is now: OK on kraken0.pmtpa.wmflabs 10.4.0.142 output: All packages OK [13:38:13] RECOVERY Current Load is now: OK on kraken1.pmtpa.wmflabs 10.4.0.240 output: OK - load average: 0.47, 0.12, 0.04 [13:38:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 185 processes [13:39:33] PROBLEM Free ram is now: UNKNOWN on kraken1.pmtpa.wmflabs 10.4.0.240 output: NRPE: Unable to read output [13:40:52] RECOVERY Disk Space is now: OK on kraken1.pmtpa.wmflabs 10.4.0.240 output: DISK OK [13:40:52] RECOVERY Total processes is now: OK on kraken1.pmtpa.wmflabs 10.4.0.240 output: PROCS OK: 83 processes [13:41:42] RECOVERY dpkg-check is now: OK on kraken1.pmtpa.wmflabs 10.4.0.240 output: All packages OK [13:50:53] PROBLEM Current Load is now: CRITICAL on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [13:51:33] PROBLEM Disk Space is now: CRITICAL on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [13:51:43] PROBLEM Total processes is now: CRITICAL on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [13:52:13] PROBLEM Free ram is now: CRITICAL on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:52:23] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:55:53] RECOVERY Current Load is now: OK on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: OK - load average: 0.18, 0.57, 0.43 [13:56:33] RECOVERY Disk Space is now: OK on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: DISK OK [13:56:43] RECOVERY Total processes is now: OK on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: PROCS OK: 93 processes [13:57:13] PROBLEM Free ram is now: UNKNOWN on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: NRPE: Unable to read output [13:57:23] RECOVERY dpkg-check is now: OK on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: All packages OK [14:03:52] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 206 processes [14:09:43] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 7.92, 6.55, 5.47 [14:11:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 217 processes [14:18:52] PROBLEM Current Load is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:19:32] PROBLEM Disk Space is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:20:12] PROBLEM Free ram is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:21:42] PROBLEM Total processes is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:22:22] PROBLEM dpkg-check is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:38:52] PROBLEM Current Load is now: CRITICAL on integration-nrpe-test.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:39:32] PROBLEM Disk Space is now: CRITICAL on integration-nrpe-test.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:40:13] PROBLEM Free ram is now: CRITICAL on integration-nrpe-test.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:41:43] PROBLEM Total processes is now: CRITICAL on integration-nrpe-test.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:42:23] PROBLEM dpkg-check is now: CRITICAL on integration-nrpe-test.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:02:13] PROBLEM Free ram is now: CRITICAL on deployment-cache-upload-test6.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [15:10:52] PROBLEM Current Load is now: CRITICAL on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [15:11:32] PROBLEM Disk Space is now: CRITICAL on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [15:12:12] PROBLEM Free ram is now: CRITICAL on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [15:13:42] PROBLEM Total processes is now: CRITICAL on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [15:14:22] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [15:18:43] RECOVERY Total processes is now: OK on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: PROCS OK: 94 processes [15:19:23] RECOVERY dpkg-check is now: OK on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: All packages OK [15:20:53] RECOVERY Current Load is now: OK on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: OK - load average: 0.18, 0.47, 0.34 [15:22:13] PROBLEM Free ram is now: UNKNOWN on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: NRPE: Unable to read output [15:26:32] RECOVERY Disk Space is now: OK on deployment-cache-upload-test7.pmtpa.wmflabs 10.4.0.100 output: DISK OK [16:04:33] PROBLEM Disk Space is now: CRITICAL on selftest2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:13] PROBLEM Free ram is now: CRITICAL on selftest2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:53] PROBLEM Current Load is now: CRITICAL on selftest2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:06:43] PROBLEM Total processes is now: CRITICAL on selftest2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:07:23] PROBLEM dpkg-check is now: CRITICAL on selftest2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:13:53] PROBLEM Current Load is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: Connection refused by host [16:14:33] PROBLEM Disk Space is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: Connection refused by host [16:15:13] PROBLEM Free ram is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:16:43] PROBLEM Total processes is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:17:23] PROBLEM dpkg-check is now: CRITICAL on kraken1.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:21:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 144 processes [16:22:42] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 159 processes [16:23:52] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 146 processes [16:25:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 188 processes [16:29:43] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 2.12, 2.68, 4.45 [16:30:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 121 processes [16:47:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 246 processes [16:49:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 232 processes [16:51:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 221 processes [17:07:43] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 4.85, 6.10, 5.39 [21:40:22] PROBLEM Free ram is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: Warning: 19% free memory [23:58:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory