[00:02:47] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 204 processes [00:07:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 142 processes [00:29:42] PROBLEM dpkg-check is now: CRITICAL on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: DPKG CRITICAL dpkg reports broken packages [00:54:33] RECOVERY dpkg-check is now: OK on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: All packages OK [01:12:23] PROBLEM Free ram is now: WARNING on aggregator1.pmtpa.wmflabs 10.4.0.79 output: Warning: 19% free memory [01:13:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 248 processes [01:17:23] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 20% free memory [01:31:23] PROBLEM Free ram is now: WARNING on aggregator1.pmtpa.wmflabs 10.4.0.79 output: Warning: 19% free memory [01:33:43] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 110.24, 69.18, 30.92 [01:34:33] PROBLEM Disk Space is now: CRITICAL on maps-tiles3.pmtpa.wmflabs 10.4.1.25 output: Connection refused by host [01:35:12] PROBLEM Free ram is now: CRITICAL on maps-tiles3.pmtpa.wmflabs 10.4.1.25 output: Connection refused by host [01:35:52] PROBLEM Current Load is now: CRITICAL on maps-tiles3.pmtpa.wmflabs 10.4.1.25 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:36:22] PROBLEM Free ram is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: Critical: 3% free memory [01:36:42] PROBLEM Total processes is now: CRITICAL on maps-tiles3.pmtpa.wmflabs 10.4.1.25 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:37:22] PROBLEM dpkg-check is now: CRITICAL on maps-tiles3.pmtpa.wmflabs 10.4.1.25 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:38:42] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 152 processes [01:41:22] RECOVERY Free ram is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: OK: 88% free memory [01:43:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 117 processes [01:58:43] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 6.75, 8.94, 17.27 [02:51:02] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:51:52] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:52:02] PROBLEM SSH is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CRITICAL - Socket timeout after 10 seconds [02:52:52] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:08:43] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 4.06, 4.32, 4.87 [08:53:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 160 processes [08:58:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 128 processes [10:16:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 186 processes [12:10:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 191 processes [12:20:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 127 processes [13:17:52] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 208 processes [14:53:52] PROBLEM Current Load is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [14:54:32] PROBLEM Disk Space is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [14:55:13] PROBLEM Free ram is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [14:56:43] PROBLEM Total processes is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [14:57:23] PROBLEM dpkg-check is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [15:09:32] PROBLEM Disk Space is now: CRITICAL on maps-tiles4.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [15:10:12] PROBLEM Free ram is now: CRITICAL on maps-tiles4.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [15:10:52] PROBLEM Current Load is now: CRITICAL on maps-tiles4.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:11:42] PROBLEM Total processes is now: CRITICAL on maps-tiles4.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:12:22] PROBLEM dpkg-check is now: CRITICAL on maps-tiles4.pmtpa.wmflabs 10.4.1.98 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:18:53] PROBLEM Current Load is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [15:19:33] PROBLEM Disk Space is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [15:20:13] PROBLEM Free ram is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [15:21:43] PROBLEM Total processes is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [15:22:23] PROBLEM dpkg-check is now: CRITICAL on he-moodle.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [15:45:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [15:48:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 189 processes [15:48:53] PROBLEM Current Load is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [15:49:33] PROBLEM Disk Space is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:50:13] PROBLEM Free ram is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:50:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 176 processes [15:51:43] PROBLEM Total processes is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:52:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-testing.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:54:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 192 processes [15:55:54] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 146 processes [16:10:53] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 150 processes [16:28:52] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 156 processes [16:30:33] PROBLEM Total processes is now: CRITICAL on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS CRITICAL: 279 processes [16:30:53] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 92% free memory [16:31:23] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 89% free memory [16:32:13] RECOVERY Free ram is now: OK on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: OK: 93% free memory [16:43:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 244 processes [16:44:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 234 processes [16:48:52] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 166 processes [16:59:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 192 processes [18:59:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 309 processes [20:24:48] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 198 processes [20:29:44] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 219 processes [20:44:44] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 195 processes [21:07:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 151 processes [21:08:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 199 processes [21:15:42] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 4.56, 5.60, 5.34 [21:17:52] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 150 processes [21:23:53] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 217 processes [21:25:43] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.59, 4.29, 4.78 [21:28:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 169 processes [21:34:42] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 318 processes [21:52:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 174 processes [21:54:52] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 156 processes [21:57:54] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 118 processes [22:14:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 246 processes [22:34:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 213 processes [22:44:42] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [22:47:43] RECOVERY Disk Space is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: DISK OK [22:47:43] RECOVERY Free ram is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: OK: 85% free memory [22:49:03] RECOVERY Current Load is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: OK - load average: 0.09, 0.27, 0.15 [22:49:13] RECOVERY Total processes is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: PROCS OK: 91 processes [22:49:33] RECOVERY dpkg-check is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: All packages OK [22:50:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 171 processes [22:50:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 151 processes [22:55:52] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 148 processes [23:09:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 216 processes [23:32:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 154 processes [23:49:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 193 processes