[00:22:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 203 processes [00:27:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [01:02:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [01:05:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 186 processes [01:10:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 98 processes [01:11:42] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 615 processes [01:13:22] PROBLEM Free ram is now: UNKNOWN on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: NRPE: Call to fork() failed [01:16:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 104 processes [01:18:23] RECOVERY Free ram is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: OK: 91% free memory [01:42:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 190 processes [02:30:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 182 processes [03:55:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 211 processes [04:00:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [04:05:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [04:59:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 262 processes [06:10:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 250 processes [06:30:22] RECOVERY dpkg-check is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: All packages OK [06:36:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 207 processes [06:43:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 185 processes [06:45:52] PROBLEM Free ram is now: CRITICAL on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: Critical: 5% free memory [06:51:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 189 processes [06:53:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 99 processes [06:55:53] PROBLEM Free ram is now: WARNING on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: Warning: 6% free memory [07:05:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 181 processes [07:09:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [07:14:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 244 processes [07:15:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 217 processes [07:15:52] PROBLEM Free ram is now: CRITICAL on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: Critical: 5% free memory [07:30:44] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 187 processes [08:29:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [08:54:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 205 processes [09:04:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 190 processes [09:15:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 163 processes [09:20:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [09:36:02] PROBLEM SSH is now: CRITICAL on upload-wizard.pmtpa.wmflabs 10.4.0.37 output: CRITICAL - Socket timeout after 10 seconds [09:51:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [09:56:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 184 processes [10:05:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 247 processes [10:39:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 229 processes [11:01:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [11:15:53] RECOVERY SSH is now: OK on upload-wizard.pmtpa.wmflabs 10.4.0.37 output: SSH OK - OpenSSH_5.8p1 Debian-7ubuntu1 (protocol 2.0) [11:26:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [11:39:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 178 processes [11:40:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [11:44:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 211 processes [11:45:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [11:49:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 184 processes [11:55:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 200 processes [11:59:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [12:29:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [12:59:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [13:13:12] RECOVERY Disk Space is now: OK on rds.pmtpa.wmflabs 10.4.0.18 output: DISK OK [13:13:22] RECOVERY Current Load is now: OK on rds.pmtpa.wmflabs 10.4.0.18 output: OK - load average: 0.82, 0.71, 0.33 [13:13:52] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [13:16:23] PROBLEM Total processes is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: PROCS WARNING: 161 processes [13:32:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [13:40:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 172 processes [13:49:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 199 processes [14:02:52] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [14:10:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [14:19:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 205 processes [14:33:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [14:35:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 187 processes [14:54:03] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 25% free memory [15:03:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [15:09:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [15:24:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 211 processes [15:34:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [15:45:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 192 processes [16:00:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 126 processes [16:04:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [16:10:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 203 processes [16:15:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 196 processes [16:18:53] PROBLEM Current Load is now: CRITICAL on oauth-apache02.pmtpa.wmflabs 10.4.1.94 output: Connection refused by host [16:19:33] PROBLEM Disk Space is now: CRITICAL on oauth-apache02.pmtpa.wmflabs 10.4.1.94 output: Connection refused by host [16:20:13] PROBLEM Free ram is now: CRITICAL on oauth-apache02.pmtpa.wmflabs 10.4.1.94 output: Connection refused by host [16:20:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 223 processes [16:21:43] PROBLEM Total processes is now: CRITICAL on oauth-apache02.pmtpa.wmflabs 10.4.1.94 output: Connection refused by host [16:22:27] PROBLEM dpkg-check is now: CRITICAL on oauth-apache02.pmtpa.wmflabs 10.4.1.94 output: Connection refused by host [16:26:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 211 processes [16:31:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [16:34:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [17:04:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [17:09:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 211 processes [17:14:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 190 processes [17:34:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [17:59:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 208 processes [18:04:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [18:16:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 214 processes [18:31:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 193 processes [18:34:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [18:35:02] PROBLEM SSH is now: CRITICAL on upload-wizard.pmtpa.wmflabs 10.4.0.37 output: CRITICAL - Socket timeout after 10 seconds [18:36:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [18:38:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 217 processes [19:03:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 181 processes [19:04:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [19:09:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 181 processes [19:19:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 208 processes [19:34:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [19:43:24] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 215 processes [19:44:44] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 184 processes [19:53:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 105 processes [20:04:54] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [20:08:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [20:16:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [20:32:12] PROBLEM dpkg-check is now: CRITICAL on puppet1.pmtpa.wmflabs 10.4.0.251 output: DPKG CRITICAL dpkg reports broken packages [20:33:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 184 processes [20:36:43] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [20:37:33] PROBLEM Total processes is now: CRITICAL on puppet1.pmtpa.wmflabs 10.4.0.251 output: PROCS CRITICAL: 264 processes [20:46:02] PROBLEM Current Load is now: WARNING on puppet1.pmtpa.wmflabs 10.4.0.251 output: WARNING - load average: 9.30, 10.19, 7.20 [20:53:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 211 processes [20:55:53] RECOVERY SSH is now: OK on upload-wizard.pmtpa.wmflabs 10.4.0.37 output: SSH OK - OpenSSH_5.8p1 Debian-7ubuntu1 (protocol 2.0) [21:04:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 205 processes [21:07:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [21:09:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 184 processes [21:15:33] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 31% free memory [21:37:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [21:39:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 208 processes [21:54:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 187 processes [22:07:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [22:21:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 194 processes [22:37:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [22:39:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 211 processes [22:51:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 211 processes [22:56:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [23:01:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 187 processes [23:01:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 178 processes [23:07:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [23:14:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 197 processes [23:34:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 214 processes [23:35:43] PROBLEM Current Load is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: WARNING - load average: 7.38, 11.71, 5.55 [23:37:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [23:40:42] RECOVERY Current Load is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK - load average: 0.40, 4.39, 4.04