[00:03:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:08:52] PROBLEM Current Load is now: CRITICAL on testing-echo.pmtpa.wmflabs 10.4.1.93 output: Connection refused by host [00:09:32] PROBLEM Disk Space is now: CRITICAL on testing-echo.pmtpa.wmflabs 10.4.1.93 output: Connection refused by host [00:10:13] PROBLEM Free ram is now: CRITICAL on testing-echo.pmtpa.wmflabs 10.4.1.93 output: Connection refused by host [00:11:43] PROBLEM Total processes is now: CRITICAL on testing-echo.pmtpa.wmflabs 10.4.1.93 output: Connection refused by host [00:11:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:12:23] PROBLEM dpkg-check is now: CRITICAL on testing-echo.pmtpa.wmflabs 10.4.1.93 output: Connection refused by host [00:23:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:33:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:41:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:45:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 198 processes [00:50:13] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 147 processes [00:53:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:58:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 279 processes [01:03:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:12:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:24:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:42:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:54:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:03:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:12:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:24:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:42:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:54:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:03:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:06:13] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 167 processes [03:12:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:16:13] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 104 processes [03:24:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:44:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:54:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:03:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:14:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:24:13] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 174 processes [04:24:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:33:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:44:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:54:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:04:02] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:14:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:24:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:34:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:44:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:55:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:02:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 183 processes [06:04:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:12:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 100 processes [06:14:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:26:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:34:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:44:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:57:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:04:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:14:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:28:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:29:12] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:44:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:04:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:14:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:28:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:44:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:04:43] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [09:04:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:14:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:24:33] RECOVERY SSH is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [09:28:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:46:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:05:17] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:17:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:28:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:35:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:47:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:05:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:17:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:25:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 172 processes [11:28:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:30:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 102 processes [11:35:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:47:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:05:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:12:23] RECOVERY dpkg-check is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: All packages OK [12:13:53] RECOVERY Current Load is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: OK - load average: 0.19, 0.15, 0.06 [12:14:33] RECOVERY Disk Space is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: DISK OK [12:15:12] PROBLEM Free ram is now: UNKNOWN on bots-gs.pmtpa.wmflabs 10.4.1.88 output: NRPE: Unable to read output [12:16:42] RECOVERY Total processes is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: PROCS OK: 89 processes [12:17:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:28:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:35:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:47:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:05:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:17:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:23:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 232 processes [13:28:13] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 174 processes [13:28:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:29:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:35:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:38:13] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 104 processes [13:47:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:58:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:59:49] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:17:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:29:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:30:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:36:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:46:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 253 processes [14:47:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:59:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:00:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:06:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:17:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:29:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:31:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:36:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:47:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:59:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:01:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:06:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:17:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:29:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:31:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:36:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:49:22] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:59:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:01:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:08:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:19:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:29:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:32:12] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:35:43] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [17:38:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:49:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:59:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:02:12] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:05:37] RECOVERY SSH is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [18:08:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:16:12] RECOVERY Total processes is now: OK on follow01d.pmtpa.wmflabs 10.4.1.40 output: PROCS OK: 91 processes [18:17:12] RECOVERY Disk Space is now: OK on follow01d.pmtpa.wmflabs 10.4.1.40 output: DISK OK [18:17:32] RECOVERY Current Load is now: OK on follow01d.pmtpa.wmflabs 10.4.1.40 output: OK - load average: 0.49, 0.36, 0.14 [18:17:52] RECOVERY Free ram is now: OK on follow01d.pmtpa.wmflabs 10.4.1.40 output: OK: 96% free memory [18:19:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:29:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:32:12] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:38:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:49:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:01:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:02:13] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:09:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:19:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:24:33] PROBLEM Disk Space is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: Connection refused by host [19:25:13] PROBLEM Free ram is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: Connection refused by host [19:25:53] PROBLEM Current Load is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: Connection refused by host [19:26:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: Connection refused by host [19:27:23] PROBLEM dpkg-check is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: Connection refused by host [19:32:32] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:32:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:39:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:49:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:59:33] RECOVERY Disk Space is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: DISK OK [20:00:12] PROBLEM Free ram is now: UNKNOWN on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: NRPE: Unable to read output [20:00:52] RECOVERY Current Load is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: OK - load average: 0.14, 0.13, 0.05 [20:01:42] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 120 processes [20:02:22] RECOVERY dpkg-check is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: All packages OK [20:02:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:03:32] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:09:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:19:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:32:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:33:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:40:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:49:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:02:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:03:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:10:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:19:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:28:43] PROBLEM SSH is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - Socket timeout after 10 seconds [21:33:32] RECOVERY Total processes is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: PROCS OK: 125 processes [21:34:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:34:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:35:43] RECOVERY dpkg-check is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: All packages OK [21:35:53] RECOVERY Disk Space is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: DISK OK [21:36:23] RECOVERY Current Load is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK - load average: 0.04, 0.03, 0.00 [21:36:43] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 89% free memory [21:38:33] RECOVERY SSH is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:41:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:49:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:04:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:04:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:11:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:19:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:34:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:34:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:41:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:49:47] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:04:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:04:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [23:11:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:19:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:34:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:34:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [23:41:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:49:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23)