[00:17:43] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 30.47, 25.39, 18.86 [01:20:53] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 155 processes [01:45:53] RECOVERY Total processes is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS OK: 148 processes [02:08:53] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 151 processes [04:36:52] PROBLEM Total processes is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS WARNING: 152 processes [04:41:53] RECOVERY Total processes is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS OK: 148 processes [06:29:34] RECOVERY dpkg-check is now: OK on worker1.pmtpa.wmflabs 10.4.1.12 output: All packages OK [06:34:23] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [06:39:22] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [06:52:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [09:15:02] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [09:37:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 158 processes [09:42:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 149 processes [11:43:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [11:47:32] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:48:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:52:33] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [13:21:52] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 200 processes [13:23:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 154 processes [13:33:57] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 126 processes [14:11:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 188 processes [15:28:54] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [15:28:54] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:29:33] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:30:13] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:31:43] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:32:23] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev2.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:33:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:58:52] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev3.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:59:32] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev3.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:00:12] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev3.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:01:42] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev3.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:02:22] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev3.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:03:52] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [16:10:13] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev4.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [16:10:53] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev4.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [16:11:33] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev4.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [16:11:43] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev4.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [16:12:24] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev4.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [16:18:52] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev5.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:19:32] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev5.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:20:13] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev5.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:21:43] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev5.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:22:23] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev5.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:43:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [16:53:52] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev7.pmtpa.wmflabs 10.4.1.2 output: Connection refused by host [16:54:32] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev7.pmtpa.wmflabs 10.4.1.2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:55:12] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev7.pmtpa.wmflabs 10.4.1.2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:56:42] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev7.pmtpa.wmflabs 10.4.1.2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:57:22] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev7.pmtpa.wmflabs 10.4.1.2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:08:53] PROBLEM Current Load is now: CRITICAL on selftest3.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:09:33] PROBLEM Disk Space is now: CRITICAL on selftest3.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:10:13] PROBLEM Free ram is now: CRITICAL on selftest3.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:11:43] PROBLEM Total processes is now: CRITICAL on selftest3.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:12:23] PROBLEM dpkg-check is now: CRITICAL on selftest3.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:23:53] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:24:33] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:25:13] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:26:43] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:27:23] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:08:53] PROBLEM Current Load is now: CRITICAL on selftest4.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:08:53] PROBLEM Disk Space is now: CRITICAL on selftest5.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [18:09:33] PROBLEM Disk Space is now: CRITICAL on selftest4.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:09:33] PROBLEM Free ram is now: CRITICAL on selftest5.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:10:12] PROBLEM Free ram is now: CRITICAL on selftest4.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:10:52] PROBLEM Total processes is now: CRITICAL on selftest5.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:11:42] PROBLEM Total processes is now: CRITICAL on selftest4.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:11:42] PROBLEM dpkg-check is now: CRITICAL on selftest5.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:12:22] PROBLEM dpkg-check is now: CRITICAL on selftest4.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:13:12] PROBLEM Current Load is now: CRITICAL on selftest5.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:13:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 168 processes [18:18:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 147 processes [19:12:42] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 13.63, 17.29, 19.26 [19:17:43] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 23.37, 20.78, 20.23 [19:18:53] PROBLEM Current Load is now: CRITICAL on testlabs-abogott-dev9.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:19:33] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott-dev9.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:20:13] PROBLEM Free ram is now: CRITICAL on testlabs-abogott-dev9.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:21:43] PROBLEM Total processes is now: CRITICAL on testlabs-abogott-dev9.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:22:23] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev9.pmtpa.wmflabs 10.4.0.251 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:27:42] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 15.75, 16.55, 18.54 [19:34:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 154 processes [19:39:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 147 processes [19:41:53] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 149 processes [19:42:13] RECOVERY Total processes is now: OK on nginx-dev1.pmtpa.wmflabs 10.4.0.68 output: PROCS OK: 98 processes [19:42:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 165 processes [20:09:53] PROBLEM host: testlabs-otto0.pmtpa.wmflabs is DOWN address: 10.4.1.110 CRITICAL - Host Unreachable (10.4.1.110) [20:13:13] RECOVERY host: testlabs-otto0.pmtpa.wmflabs is UP address: 10.4.1.110 PING OK - Packet loss = 0%, RTA = 0.75 ms [20:13:43] PROBLEM Total processes is now: CRITICAL on testlabs-otto0.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [20:14:23] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 5% free memory [20:14:23] PROBLEM dpkg-check is now: CRITICAL on testlabs-otto0.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [20:15:54] PROBLEM Current Load is now: CRITICAL on testlabs-otto0.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [20:16:34] PROBLEM Disk Space is now: CRITICAL on testlabs-otto0.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [20:17:14] PROBLEM Free ram is now: CRITICAL on testlabs-otto0.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [20:29:23] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [20:44:23] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [20:46:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [20:50:53] PROBLEM Current Load is now: CRITICAL on testlabs-otto1.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [20:51:33] PROBLEM Disk Space is now: CRITICAL on testlabs-otto1.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [20:52:13] PROBLEM Free ram is now: CRITICAL on testlabs-otto1.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [20:52:23] PROBLEM dpkg-check is now: CRITICAL on testlabs-otto1.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [20:53:43] PROBLEM Total processes is now: CRITICAL on testlabs-otto1.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [20:55:12] PROBLEM Free ram is now: UNKNOWN on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: NRPE: Unable to read output [20:56:42] RECOVERY Total processes is now: OK on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: PROCS OK: 79 processes [20:57:23] RECOVERY dpkg-check is now: OK on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: All packages OK [20:58:52] RECOVERY Current Load is now: OK on testlabs-abogott-dev8.pmtpa.wmflabs 10.4.1.107 output: OK - load average: 0.02, 0.15, 0.10 [21:29:22] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [21:47:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 390 processes [21:49:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 365 processes [21:52:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 408 processes [21:53:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [21:55:42] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 32.23, 25.98, 12.80 [21:58:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [21:59:42] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 141 processes [22:19:52] PROBLEM host: testlabs-otto2.pmtpa.wmflabs is DOWN address: 10.4.1.109 CRITICAL - Host Unreachable (10.4.1.109) [22:24:33] RECOVERY host: testlabs-otto2.pmtpa.wmflabs is UP address: 10.4.1.109 PING OK - Packet loss = 0%, RTA = 0.63 ms [22:25:52] PROBLEM Total processes is now: CRITICAL on testlabs-otto2.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [22:25:52] PROBLEM dpkg-check is now: CRITICAL on testlabs-otto2.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [22:27:12] PROBLEM Current Load is now: CRITICAL on testlabs-otto2.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [22:28:02] PROBLEM Disk Space is now: CRITICAL on testlabs-otto2.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:28:42] PROBLEM Free ram is now: CRITICAL on testlabs-otto2.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:28:52] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [22:46:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [23:08:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory