[00:01:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 217 processes [00:07:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [00:26:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 187 processes [00:29:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 217 processes [00:37:52] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [00:46:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 26% free memory [01:01:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [01:07:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 391 processes [01:07:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [01:09:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 197 processes [01:17:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 105 processes [01:24:45] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 214 processes [01:29:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 193 processes [01:37:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [02:09:12] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [02:11:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 199 processes [02:39:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [02:42:22] PROBLEM Free ram is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: Warning: 19% free memory [02:59:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 217 processes [03:09:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [03:09:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 187 processes [03:16:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [03:21:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 187 processes [03:39:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [03:41:43] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [04:09:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [04:39:15] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [04:54:22] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [05:09:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [05:24:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 202 processes [05:30:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory [05:34:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [05:42:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [06:12:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [06:36:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 201 processes [06:41:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [06:42:52] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [06:55:42] PROBLEM Total processes is now: CRITICAL on master.pmtpa.wmflabs 10.4.0.14 output: PROCS CRITICAL: 204 processes [07:09:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 202 processes [07:12:52] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [07:14:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [07:39:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 202 processes [07:42:52] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [07:44:46] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 188 processes [08:12:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [08:44:12] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [08:46:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [09:05:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 165 processes [09:10:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 110 processes [09:14:12] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [09:24:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 229 processes [09:26:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 223 processes [09:44:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [09:49:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [10:04:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 232 processes [10:14:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [10:29:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [10:31:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 178 processes [10:44:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [11:08:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 177 processes [11:13:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 102 processes [11:14:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [11:44:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [11:51:42] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [12:07:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [12:14:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [12:47:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [13:17:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [13:22:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 208 processes [13:37:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 169 processes [13:48:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [14:07:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [14:19:12] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [14:42:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 178 processes [14:49:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [15:04:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 203 processes [15:09:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 193 processes [15:19:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [15:49:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [16:19:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [16:44:03] PROBLEM SSH is now: CRITICAL on upload-wizard.pmtpa.wmflabs 10.4.0.37 output: CRITICAL - Socket timeout after 10 seconds [16:49:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [16:56:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 193 processes [17:19:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [17:24:42] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 196 processes [17:34:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 241 processes [17:36:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 205 processes [17:41:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 172 processes [17:44:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 194 processes [17:49:53] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [17:56:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 211 processes [18:06:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [18:11:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [18:20:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [18:25:15] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [18:29:43] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 223 processes [18:30:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [18:50:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [18:55:13] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [19:20:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [19:40:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:52:22] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [20:01:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 172 processes [20:06:43] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 214 processes [20:21:43] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 178 processes [20:22:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [20:25:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [20:52:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [21:21:43] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [21:22:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [21:52:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [22:15:12] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [22:20:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [22:22:32] PROBLEM SSH is now: CRITICAL on ganglia-test2.pmtpa.wmflabs 10.4.0.157 output: CRITICAL - Socket timeout after 10 seconds [22:23:02] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [22:50:13] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [22:53:23] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [23:14:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 175 processes [23:24:12] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8) [23:24:42] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 202 processes [23:30:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [23:44:42] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 166 processes [23:54:13] PROBLEM host: stackfarm-sql1.pmtpa.wmflabs is DOWN address: 10.4.1.8 CRITICAL - Host Unreachable (10.4.1.8)