[00:15:53] PROBLEM Disk Space is now: CRITICAL on build-precise1.pmtpa.wmflabs 10.4.0.173 output: DISK CRITICAL - free space: / 0 MB (0% inode=48%): [00:17:13] RECOVERY Free ram is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: OK: 95% free memory [00:20:52] RECOVERY Disk Space is now: OK on build-precise1.pmtpa.wmflabs 10.4.0.173 output: DISK OK [00:53:52] PROBLEM Disk Space is now: CRITICAL on build-precise1.pmtpa.wmflabs 10.4.0.173 output: DISK CRITICAL - free space: / 108 MB (1% inode=59%): [01:16:42] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 262 processes [01:26:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 124 processes [02:18:42] PROBLEM host: testing-newimage13.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:31:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 189 processes [02:37:53] PROBLEM Disk Space is now: CRITICAL on build-precise1.pmtpa.wmflabs 10.4.0.173 output: DISK CRITICAL - free space: / 0 MB (0% inode=64%): [02:41:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 194 processes [02:45:32] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 5.30, 9.63, 17.22 [02:57:53] RECOVERY Disk Space is now: OK on build-precise1.pmtpa.wmflabs 10.4.0.173 output: DISK OK [03:44:03] PROBLEM host: testing-newimage14.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:14:32] PROBLEM host: testing-newimage14.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:21:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 201 processes [04:41:43] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [04:44:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 168 processes [04:45:32] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 4.64, 4.55, 4.84 [04:46:02] PROBLEM host: testing-newimage14.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:58:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 4.90, 5.12, 5.09 [05:13:33] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 4.48, 4.69, 4.96 [05:17:03] PROBLEM host: testing-newimage14.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:29:32] PROBLEM Disk Space is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [05:30:12] PROBLEM Free ram is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [05:30:52] PROBLEM Current Load is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.1.102 output: Connection refused by host [05:31:42] PROBLEM Total processes is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:32:22] PROBLEM dpkg-check is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.1.102 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:08:53] PROBLEM Current Load is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:09:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:10:13] PROBLEM Free ram is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:11:43] PROBLEM Total processes is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:12:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage15.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:14:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 5.36, 5.62, 5.32 [06:16:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 152 processes [06:18:52] PROBLEM Current Load is now: CRITICAL on testing-newimage18.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [06:18:52] PROBLEM Disk Space is now: CRITICAL on testing-newimage19.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [06:18:52] PROBLEM Free ram is now: CRITICAL on testing-newimage20.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [06:19:32] PROBLEM Disk Space is now: CRITICAL on testing-newimage18.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [06:19:32] PROBLEM Free ram is now: CRITICAL on testing-newimage19.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [06:20:13] PROBLEM Free ram is now: CRITICAL on testing-newimage18.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [06:20:13] PROBLEM Total processes is now: CRITICAL on testing-newimage20.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [06:20:53] PROBLEM Total processes is now: CRITICAL on testing-newimage22.pmtpa.wmflabs 10.4.1.111 output: Connection refused by host [06:21:03] PROBLEM Total processes is now: CRITICAL on testing-newimage19.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:21:03] PROBLEM dpkg-check is now: CRITICAL on testing-newimage20.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [06:21:33] PROBLEM Total processes is now: CRITICAL on testing-newimage21.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [06:21:33] PROBLEM dpkg-check is now: CRITICAL on testing-newimage22.pmtpa.wmflabs 10.4.1.111 output: Connection refused by host [06:21:43] PROBLEM Total processes is now: CRITICAL on testing-newimage18.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:21:43] PROBLEM dpkg-check is now: CRITICAL on testing-newimage19.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:21:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 251 processes [06:21:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 225 processes [06:22:13] PROBLEM dpkg-check is now: CRITICAL on testing-newimage21.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [06:22:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage18.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:22:23] PROBLEM Current Load is now: CRITICAL on testing-newimage20.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [06:22:23] PROBLEM Free ram is now: CRITICAL on testing-newimage22.pmtpa.wmflabs 10.4.1.111 output: Connection refused by host [06:22:53] PROBLEM Current Load is now: CRITICAL on testing-newimage19.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:23:03] PROBLEM Current Load is now: CRITICAL on testing-newimage22.pmtpa.wmflabs 10.4.1.111 output: Connection refused by host [06:23:13] PROBLEM Disk Space is now: CRITICAL on testing-newimage20.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:23:13] PROBLEM Free ram is now: CRITICAL on testing-newimage21.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [06:23:43] PROBLEM Current Load is now: CRITICAL on testing-newimage21.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [06:23:43] PROBLEM Disk Space is now: CRITICAL on testing-newimage22.pmtpa.wmflabs 10.4.1.111 output: Connection refused by host [06:24:23] PROBLEM Disk Space is now: CRITICAL on testing-newimage21.pmtpa.wmflabs 10.4.1.110 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:24:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [06:26:45] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 169 processes [06:26:45] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 182 processes [06:29:35] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 191 processes [07:02:42] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 6.15, 6.23, 5.97 [07:04:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [07:26:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 213 processes [07:29:43] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 167 processes [07:31:48] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 659 processes [07:35:52] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 9.77, 9.35, 6.61 [07:44:42] RECOVERY Total processes is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS OK: 144 processes [07:48:18] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 7.00, 6.78, 5.77 [08:46:02] PROBLEM Current Load is now: CRITICAL on puppet1.pmtpa.wmflabs 10.4.0.251 output: CRITICAL - load average: 22.32, 22.45, 20.77 [08:49:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 162 processes [08:51:03] PROBLEM Current Load is now: WARNING on puppet1.pmtpa.wmflabs 10.4.0.251 output: WARNING - load average: 9.76, 15.47, 18.39 [08:51:43] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [08:51:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 156 processes [08:53:23] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 0.09, 2.16, 4.79 [09:05:53] RECOVERY Current Load is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: OK - load average: 1.74, 1.95, 4.26 [09:09:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 223 processes [09:09:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 189 processes [09:14:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 222 processes [09:16:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 204 processes [09:17:52] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 172 processes [09:19:52] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [09:22:47] PROBLEM Total processes is now: CRITICAL on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS CRITICAL: 201 processes [09:24:57] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [09:26:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 189 processes [09:32:48] RECOVERY Total processes is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS OK: 128 processes [09:39:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 177 processes [10:19:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 247 processes [10:54:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [10:57:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 206 processes [10:59:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:01:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 219 processes [11:09:33] PROBLEM Disk Space is now: CRITICAL on testing-newimage23.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [11:10:12] PROBLEM Free ram is now: CRITICAL on testing-newimage23.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:10:52] PROBLEM Current Load is now: CRITICAL on testing-newimage23.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:11:42] PROBLEM Total processes is now: CRITICAL on testing-newimage23.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:12:22] PROBLEM dpkg-check is now: CRITICAL on testing-newimage23.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:19:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 173 processes [11:44:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 203 processes [11:59:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [12:12:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [12:24:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:04:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [13:12:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 200 processes [13:22:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [13:27:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 197 processes [13:29:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [14:14:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [14:39:54] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [14:49:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [15:07:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 209 processes [15:17:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 197 processes [15:29:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [16:01:32] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 195 processes [16:06:33] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [16:16:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 156 processes [16:24:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 203 processes [16:31:48] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 324 processes [16:32:55] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 33.28, 21.00, 14.50 [16:39:58] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [17:22:43] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 218 processes [17:26:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 296 processes [17:32:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 166 processes [17:37:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 98 processes [17:49:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [17:54:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:04:52] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [19:24:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [20:24:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [20:44:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [21:04:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [22:24:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 8% free memory [22:44:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 182 processes [22:49:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 98 processes