[00:00:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [00:24:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [00:25:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [00:29:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 205 processes [00:49:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 199 processes [00:54:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [01:10:42] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 535 processes [01:15:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 103 processes [01:24:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [01:39:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 238 processes [01:39:53] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [01:44:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 190 processes [01:50:13] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [01:54:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [01:55:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [01:59:43] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 145 processes [02:24:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [02:55:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [03:27:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [03:32:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 163 processes [03:42:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 208 processes [03:57:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [04:27:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [04:50:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 181 processes [04:57:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [04:59:23] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [05:25:42] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 142 processes [05:27:52] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [05:41:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 160 processes [05:57:52] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [06:06:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [06:25:15] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [06:26:45] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [06:27:55] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [06:30:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [06:31:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 178 processes [06:49:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 235 processes [06:51:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [06:59:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [07:05:13] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [07:29:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [07:36:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 184 processes [07:45:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 8% free memory [07:59:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [08:16:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [08:19:12] RECOVERY host: stackfarm-sql1.pmtpa.wmflabs is UP address: 10.4.1.8 PING OK - Packet loss = 0%, RTA = 0.78 ms [08:36:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [08:41:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 214 processes [08:41:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [09:06:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [09:14:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 202 processes [09:19:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 199 processes [09:24:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 214 processes [09:26:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [09:31:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 175 processes [09:46:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 163 processes [09:55:13] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [10:00:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:11:42] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 145 processes [11:19:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 167 processes [11:29:43] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 145 processes [12:02:23] PROBLEM dpkg-check is now: UNKNOWN on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: NRPE: Call to fork() failed [12:03:23] PROBLEM Total processes is now: UNKNOWN on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: NRPE: Call to fork() failed [12:03:43] PROBLEM Disk Space is now: UNKNOWN on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: NRPE: Call to fork() failed [12:07:23] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:07:33] PROBLEM Current Load is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:08:23] PROBLEM Total processes is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:08:43] PROBLEM Disk Space is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:10:52] PROBLEM SSH is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Server answer: [12:11:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 176 processes [12:12:32] RECOVERY Current Load is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK - load average: 1.81, 1.43, 1.21 [12:13:42] RECOVERY Disk Space is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: DISK OK [12:14:22] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 24% free memory [12:15:53] RECOVERY SSH is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [12:16:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 209 processes [12:17:23] RECOVERY dpkg-check is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: All packages OK [12:26:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 120 processes [12:58:33] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:58:43] PROBLEM Total processes is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [13:03:32] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [13:03:42] RECOVERY Total processes is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: PROCS OK: 230 processes [13:07:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [13:17:42] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [13:53:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 151 processes [14:43:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 175 processes [15:20:12] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [15:22:22] PROBLEM Free ram is now: WARNING on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Warning: 19% free memory [15:25:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:53:53] PROBLEM Current Load is now: CRITICAL on solr-wlm2.pmtpa.wmflabs 10.4.1.97 output: Connection refused by host [15:54:33] PROBLEM Disk Space is now: CRITICAL on solr-wlm2.pmtpa.wmflabs 10.4.1.97 output: Connection refused by host [15:55:13] PROBLEM Free ram is now: CRITICAL on solr-wlm2.pmtpa.wmflabs 10.4.1.97 output: Connection refused by host [15:56:43] PROBLEM Total processes is now: CRITICAL on solr-wlm2.pmtpa.wmflabs 10.4.1.97 output: Connection refused by host [15:57:23] PROBLEM dpkg-check is now: CRITICAL on solr-wlm2.pmtpa.wmflabs 10.4.1.97 output: Connection refused by host [16:13:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 211 processes [16:16:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [16:18:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 194 processes [16:21:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [17:02:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 154 processes [17:12:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 102 processes [17:14:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [17:33:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 214 processes [17:34:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 247 processes [17:36:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 214 processes [17:45:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [17:48:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 169 processes [17:49:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [17:50:12] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [17:55:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [17:55:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 166 processes [17:56:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [18:16:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [18:29:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 211 processes [18:46:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [19:09:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 187 processes [19:16:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [19:24:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 217 processes [19:35:12] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [19:36:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 184 processes [19:40:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:46:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [20:17:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [20:25:13] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [20:47:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [20:54:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [21:00:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [21:17:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [21:41:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 189 processes [21:47:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [21:51:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 244 processes [21:56:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 121 processes [22:17:23] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 5% free memory [22:17:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [22:47:42] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:17:43] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:44:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 213 processes [23:48:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:49:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 106 processes