[00:23:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 512 processes [00:45:32] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 218 processes [00:50:32] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 169 processes [00:53:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 191 processes [00:55:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 201 processes [01:10:42] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 386 processes [01:20:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 143 processes [01:23:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 467 processes [01:34:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 194 processes [01:36:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 200 processes [01:39:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 275 processes [01:41:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 245 processes [01:43:46] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 188 processes [02:07:52] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK CRITICAL - free space: / 255 MB (2% inode=72%): [02:18:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 724 processes [03:03:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 191 processes [03:23:32] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 142 processes [03:45:32] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 27.22, 24.98, 21.10 [04:30:32] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 15.54, 17.15, 19.79 [04:46:33] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 230 processes [04:51:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 168 processes [04:53:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 416 processes [05:10:33] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 21.25, 22.29, 20.48 [05:30:32] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 14.96, 16.03, 17.40 [06:24:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 165 processes [06:29:45] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS CRITICAL: 203 processes [06:34:33] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 103 processes [06:54:43] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 199 processes [11:55:32] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 21.07, 21.92, 20.41 [12:10:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 17.73, 17.48, 18.88 [12:25:32] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 29.28, 22.15, 20.23 [14:12:13] PROBLEM Free ram is now: WARNING on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Warning: 19% free memory [14:20:32] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 31.81, 21.21, 19.54 [14:25:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 184 processes [14:30:33] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 119 processes [15:15:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 9.57, 16.79, 19.88 [15:20:32] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 21.16, 19.76, 20.30 [16:03:33] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 204 processes [16:13:32] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 107 processes [16:25:32] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 11.92, 15.01, 19.17 [16:31:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [16:33:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 183 processes [16:44:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [16:48:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 210 processes [16:49:38] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 205 processes [16:49:38] PROBLEM dpkg-check is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Socket timeout after 10 seconds. [16:52:33] PROBLEM Disk Space is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Socket timeout after 10 seconds. [16:52:33] PROBLEM Free ram is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Socket timeout after 10 seconds. [16:53:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 192 processes [16:54:03] PROBLEM Current Load is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Socket timeout after 10 seconds. [16:54:03] PROBLEM Total processes is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: CHECK_NRPE: Socket timeout after 10 seconds. [16:56:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 204 processes [17:36:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [18:08:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 201 processes [18:34:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 187 processes [18:59:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 214 processes [19:29:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [20:04:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 237 processes [20:10:53] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [20:39:33] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 156 processes [20:40:57] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 28% free memory [20:44:32] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 145 processes [20:58:53] PROBLEM Current Load is now: CRITICAL on asher-m1.pmtpa.wmflabs 10.4.0.10 output: Connection refused by host [20:59:33] PROBLEM Disk Space is now: CRITICAL on asher-m1.pmtpa.wmflabs 10.4.0.10 output: Connection refused by host [21:00:13] PROBLEM Free ram is now: CRITICAL on asher-m1.pmtpa.wmflabs 10.4.0.10 output: Connection refused by host [21:01:43] PROBLEM Total processes is now: CRITICAL on asher-m1.pmtpa.wmflabs 10.4.0.10 output: Connection refused by host [21:02:23] PROBLEM dpkg-check is now: CRITICAL on asher-m1.pmtpa.wmflabs 10.4.0.10 output: Connection refused by host [21:21:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 204 processes [21:23:53] PROBLEM Current Load is now: CRITICAL on rt-testing6.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [21:24:33] PROBLEM Disk Space is now: CRITICAL on rt-testing6.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [21:25:13] PROBLEM Free ram is now: CRITICAL on rt-testing6.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [21:26:43] PROBLEM Total processes is now: CRITICAL on rt-testing6.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [21:27:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing6.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [21:31:22] RECOVERY Total processes is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: PROCS OK: 83 processes [21:31:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 198 processes [21:36:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 181 processes [21:39:33] PROBLEM Total processes is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [21:40:53] PROBLEM Current Load is now: CRITICAL on testing-newimage3.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [21:41:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage3.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [21:41:43] PROBLEM Total processes is now: CRITICAL on testing-newimage3.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [21:42:13] PROBLEM Free ram is now: CRITICAL on testing-newimage3.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [21:42:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage3.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [21:43:03] PROBLEM SSH is now: CRITICAL on testing-newimage3.pmtpa.wmflabs 10.4.1.99 output: Connection refused [21:51:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 213 processes [22:01:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 204 processes [22:03:53] PROBLEM Current Load is now: CRITICAL on rt-testing7.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [22:04:33] PROBLEM Disk Space is now: CRITICAL on rt-testing7.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [22:05:12] PROBLEM Free ram is now: CRITICAL on rt-testing7.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [22:06:42] PROBLEM Total processes is now: CRITICAL on rt-testing7.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [22:07:22] PROBLEM dpkg-check is now: CRITICAL on rt-testing7.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [22:09:42] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [22:14:03] PROBLEM host: testing-newimage3.pmtpa.wmflabs is DOWN address: 10.4.1.99 CRITICAL - Host Unreachable (10.4.1.99) [22:19:13] RECOVERY host: testing-newimage3.pmtpa.wmflabs is UP address: 10.4.1.99 PING OK - Packet loss = 0%, RTA = 2.61 ms [22:21:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 198 processes [22:33:03] RECOVERY SSH is now: OK on testing-newimage3.pmtpa.wmflabs 10.4.1.99 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:56:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 216 processes [23:08:52] PROBLEM Current Load is now: CRITICAL on rt-testing8.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [23:09:32] PROBLEM Disk Space is now: CRITICAL on rt-testing8.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [23:10:13] PROBLEM Free ram is now: CRITICAL on rt-testing8.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [23:11:43] PROBLEM Total processes is now: CRITICAL on rt-testing8.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [23:12:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing8.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:21:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 198 processes [23:51:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 201 processes