[00:18:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [00:48:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [01:05:13] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [01:08:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 292 processes [01:09:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 203 processes [01:10:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [01:18:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [01:18:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 148 processes [01:21:33] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 7.08, 7.92, 6.02 [01:21:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 238 processes [01:29:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 179 processes [01:31:32] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.14, 3.73, 4.70 [01:31:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 157 processes [01:48:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [02:00:13] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [02:06:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [02:19:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [02:27:43] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:27:43] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:28:53] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:30:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [02:31:13] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:31:53] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [02:41:12] RECOVERY Total processes is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: PROCS OK: 101 processes [02:41:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 175 processes [02:41:52] RECOVERY SSH is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [02:42:42] RECOVERY Disk Space is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: DISK OK [02:43:52] RECOVERY dpkg-check is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: All packages OK [02:47:43] RECOVERY Current Load is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: OK - load average: 0.34, 0.68, 0.89 [02:49:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [03:19:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [03:21:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [03:22:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 161 processes [03:27:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 125 processes [03:27:23] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [03:30:42] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:30:42] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:31:52] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:32:22] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:34:12] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:34:52] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [03:49:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [04:16:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [04:19:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [04:20:12] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [04:21:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [04:25:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:31:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 181 processes [04:51:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [05:05:13] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [05:09:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [05:21:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [05:34:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 187 processes [05:40:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [05:51:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [06:05:12] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [06:17:23] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [06:22:22] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [06:22:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [06:26:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 201 processes [06:31:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 186 processes [06:34:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 228 processes [06:35:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [06:36:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 204 processes [06:44:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 159 processes [06:49:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 135 processes [06:52:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [06:56:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 184 processes [07:09:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 187 processes [07:22:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [07:51:42] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 142 processes [07:53:22] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [08:22:23] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [08:23:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [08:27:22] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [08:42:23] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [08:44:43] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [08:49:42] RECOVERY Disk Space is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: DISK OK [08:53:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [09:23:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [09:32:22] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [09:44:44] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 157 processes [09:53:23] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [10:14:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [10:23:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [10:24:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 196 processes [10:44:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [10:54:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [11:25:22] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [11:39:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 185 processes [11:56:22] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [12:04:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 203 processes [12:26:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [12:56:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [13:09:45] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 187 processes [13:27:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [13:57:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [14:08:12] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [14:12:46] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [14:13:13] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [14:14:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 229 processes [14:19:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 178 processes [14:27:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [14:27:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 154 processes [14:28:13] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [14:31:52] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:32:42] PROBLEM Disk Space is now: CRITICAL on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: CHECK_NRPE: Socket timeout after 10 seconds. [14:32:52] PROBLEM Current Load is now: CRITICAL on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: CHECK_NRPE: Socket timeout after 10 seconds. [14:36:02] PROBLEM dpkg-check is now: CRITICAL on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: CHECK_NRPE: Socket timeout after 10 seconds. [14:36:32] PROBLEM SSH is now: CRITICAL on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: CRITICAL - Socket timeout after 10 seconds [14:36:52] RECOVERY dpkg-check is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: All packages OK [14:37:12] RECOVERY Total processes is now: OK on master.pmtpa.wmflabs 10.4.0.14 output: PROCS OK: 113 processes [14:37:42] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [14:39:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [14:42:13] PROBLEM Total processes is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [14:44:43] PROBLEM Total processes is now: UNKNOWN on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: NRPE: Call to fork() failed [14:49:43] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:49:53] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:51:32] PROBLEM Current Load is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: WARNING - load average: 9.10, 8.75, 5.90 [14:52:12] RECOVERY Total processes is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: PROCS OK: 98 processes [14:54:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 208 processes [14:54:42] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 196 processes [14:54:52] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [14:54:52] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 33% free memory [14:57:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [15:18:13] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Critical: 1% free memory [15:27:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [15:38:53] PROBLEM Current Load is now: CRITICAL on rt-testing4.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [15:39:33] PROBLEM Disk Space is now: CRITICAL on rt-testing4.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [15:40:13] PROBLEM Free ram is now: CRITICAL on rt-testing4.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [15:41:43] PROBLEM Total processes is now: CRITICAL on rt-testing4.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [15:42:23] PROBLEM dpkg-check is now: CRITICAL on rt-testing4.pmtpa.wmflabs 10.4.1.98 output: Connection refused by host [15:49:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 178 processes [15:57:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [16:04:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 120 processes [16:27:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [16:57:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [17:27:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [17:30:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 235 processes [17:35:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 131 processes [17:49:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 171 processes [17:57:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [18:06:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 205 processes [18:24:44] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 237 processes [18:28:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [18:39:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 176 processes [18:44:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 206 processes [18:46:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 207 processes [18:51:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 180 processes [18:54:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [18:58:32] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [19:29:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [19:54:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 203 processes [19:59:13] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [20:31:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [20:46:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 201 processes [20:51:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 183 processes [21:01:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 159 processes [21:01:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [21:11:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 202 processes [21:21:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 165 processes [21:31:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [22:01:53] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [22:03:13] RECOVERY Free ram is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: OK: 96% free memory [22:18:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 196 processes [22:32:22] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [22:34:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 218 processes [22:38:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [22:43:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 187 processes [22:58:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [23:02:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:15:42] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 188 processes [23:32:33] PROBLEM host: integration-jobbuilder.pmtpa.wmflabs is DOWN address: 10.4.0.21 CRITICAL - Host Unreachable (10.4.0.21) [23:39:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 212 processes [23:44:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 187 processes [23:48:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes