[00:05:46] PROBLEM Current Load is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: CRITICAL - load average: 103.99, 118.51, 71.58 [00:09:46] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:12:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:13:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:13:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:26:15] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [00:26:45] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 91.01, 243.41, 151.14 [00:40:24] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:42:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:43:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:43:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:56:15] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [01:11:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:13:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:13:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:14:03] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:26:53] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [01:34:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 187 processes [01:42:16] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:43:46] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:44:16] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:44:26] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:54:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 350 processes [01:56:55] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [02:12:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:12:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:14:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:14:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:14:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:15:14] PROBLEM Current Load is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:15:54] PROBLEM Disk Space is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:17:44] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [02:20:15] RECOVERY Current Load is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK - load average: 0.14, 0.38, 0.45 [02:20:55] RECOVERY Disk Space is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: DISK OK [02:28:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [02:29:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 197 processes [02:42:34] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:44:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:44:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:44:34] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:58:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [03:04:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 885 processes [03:12:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:14:13] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:14:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:14:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:28:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [03:43:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:44:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:44:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:44:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:53:14] PROBLEM Free ram is now: WARNING on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [03:58:14] RECOVERY Free ram is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: OK: 20% free memory [03:58:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [04:04:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 193 processes [04:14:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:14:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:14:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:14:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:19:34] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 4658 processes [04:28:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [04:39:16] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 22% free memory [04:44:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [04:44:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:44:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:44:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:44:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:52:15] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 17% free memory [04:58:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [05:14:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:15:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:15:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:17:05] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:27:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:28:12] PROBLEM Current Load is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:28:42] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [05:32:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [05:33:13] RECOVERY Current Load is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK - load average: 0.12, 0.28, 0.39 [05:44:54] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:46:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:48:03] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:48:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:59:05] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [06:16:56] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:17:16] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:18:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:18:56] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:29:06] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [06:32:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 183 processes [06:37:46] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 1067 processes [06:43:14] PROBLEM Total processes is now: CRITICAL on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS CRITICAL: 203 processes [06:47:03] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:47:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:48:13] PROBLEM Total processes is now: WARNING on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS WARNING: 194 processes [06:48:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:49:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:49:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 1057 processes [07:00:33] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [07:07:16] PROBLEM Current Load is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:07:26] PROBLEM Disk Space is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:07:36] PROBLEM dpkg-check is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:07:56] PROBLEM Total processes is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:10:13] PROBLEM SSH is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: Server answer: [07:10:43] PROBLEM Free ram is now: CRITICAL on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:12:23] PROBLEM Disk Space is now: UNKNOWN on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: NRPE: Call to fork() failed [07:12:23] RECOVERY Current Load is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: OK - load average: 9.25, 8.36, 4.58 [07:15:14] RECOVERY SSH is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [07:15:44] RECOVERY Free ram is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: OK: 47% free memory [07:17:24] RECOVERY Disk Space is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: DISK OK [07:17:34] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:17:44] RECOVERY dpkg-check is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: All packages OK [07:17:54] RECOVERY Total processes is now: OK on wikidata-repotest.pmtpa.wmflabs 10.4.0.224 output: PROCS OK: 104 processes [07:18:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:19:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:19:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:31:12] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [07:34:52] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 14% free memory [07:44:54] RECOVERY Free ram is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK: 21% free memory [07:47:35] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:47:45] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [07:48:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:49:25] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:49:25] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:52:45] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [07:52:45] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [07:54:35] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 196 processes [08:01:14] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [08:06:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 158 processes [08:11:24] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 104 processes [08:12:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 8480 processes [08:19:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:19:34] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:19:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:19:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:24:34] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 4464 processes [08:27:45] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [08:31:14] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [08:37:24] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 23% free memory [08:37:54] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 7% free memory [08:45:23] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [08:49:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:49:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:50:24] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:53:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:57:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [09:01:22] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [09:04:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 193 processes [09:12:46] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 7881 processes [09:19:46] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:19:56] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:20:24] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:22:44] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [09:23:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:31:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [09:37:47] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [09:39:37] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 342 processes [09:50:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:50:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:51:55] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:53:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:01:52] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [10:07:47] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 430 processes [10:20:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:20:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:21:55] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:23:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:31:54] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [10:37:16] PROBLEM Free ram is now: WARNING on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [10:50:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:50:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:51:59] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:52:15] RECOVERY Free ram is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: OK: 21% free memory [10:53:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:03:44] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [11:10:54] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 0.03, 1.22, 16.76 [11:11:44] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 3.37, 4.61, 16.62 [11:17:44] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [11:17:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [11:20:54] PROBLEM Disk Space is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:20:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:22:34] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:22:44] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:23:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:24:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:25:55] RECOVERY Disk Space is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: DISK OK [11:30:52] RECOVERY Current Load is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: OK - load average: 1.41, 0.75, 4.91 [11:32:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 442 processes [11:34:02] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [11:36:45] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 0.90, 1.46, 4.95 [11:50:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:53:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:53:34] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:54:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:59:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 181 processes [12:03:52] PROBLEM Current Load is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: CRITICAL - load average: 86.71, 61.49, 26.60 [12:04:02] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [12:04:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 118 processes [12:04:42] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 30.16, 42.45, 21.99 [12:08:47] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [12:16:16] PROBLEM Free ram is now: WARNING on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [12:20:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:23:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:24:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:24:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:26:15] RECOVERY Free ram is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: OK: 20% free memory [12:34:04] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [12:39:46] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [12:40:22] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 23% free memory [12:44:42] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 4.01, 9.07, 18.31 [12:48:23] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [12:48:53] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 2.09, 3.41, 11.89 [12:51:03] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:53:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:54:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:54:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:05:26] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [13:10:54] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [13:22:04] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:23:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:24:34] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:24:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:35:26] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [13:41:04] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [13:52:17] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:53:46] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:54:36] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:57:46] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:05:26] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [14:11:05] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [14:17:44] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 16.19, 14.03, 8.07 [14:21:52] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 15.84, 12.41, 7.50 [14:22:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:24:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:24:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:28:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:35:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [14:41:52] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [14:42:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 256 processes [14:47:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 165 processes [14:52:26] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 101 processes [14:52:35] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:54:45] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:54:55] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:58:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:05:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [15:11:52] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [15:17:44] PROBLEM Current Load is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: CRITICAL - load average: 30.00, 25.84, 18.80 [15:21:54] PROBLEM Current Load is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: CRITICAL - load average: 38.34, 22.78, 17.05 [15:24:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:24:54] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:25:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:26:54] PROBLEM Current Load is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: WARNING - load average: 4.03, 15.21, 15.90 [15:27:44] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 1.69, 16.52, 19.54 [15:29:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:35:34] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [15:41:52] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [15:54:45] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:55:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:56:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:57:44] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 1.29, 1.51, 4.08 [15:59:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:06:06] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [16:11:53] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [16:25:25] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:25:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:26:55] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:29:25] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:34:22] PROBLEM host: wsexport.pmtpa.wmflabs is DOWN address: 10.4.0.231 CRITICAL - Host Unreachable (10.4.0.231) [16:35:45] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 19% free memory [16:36:15] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [16:38:15] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 23% free memory [16:40:53] RECOVERY Free ram is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK: 20% free memory [16:41:53] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [16:48:53] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 18% free memory [16:55:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:56:24] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [16:58:28] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:58:28] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:59:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:04:54] PROBLEM host: wsexport.pmtpa.wmflabs is DOWN address: 10.4.0.231 CRITICAL - Host Unreachable (10.4.0.231) [17:06:44] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [17:11:55] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [17:25:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:28:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:29:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:29:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:34:54] PROBLEM host: wsexport.pmtpa.wmflabs is DOWN address: 10.4.0.231 CRITICAL - Host Unreachable (10.4.0.231) [17:36:54] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [17:42:13] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [17:55:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:56:15] PROBLEM Free ram is now: WARNING on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Warning: 19% free memory [17:58:25] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:59:25] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:59:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:01:13] RECOVERY Free ram is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: OK: 21% free memory [18:05:34] PROBLEM host: wsexport.pmtpa.wmflabs is DOWN address: 10.4.0.231 CRITICAL - Host Unreachable (10.4.0.231) [18:06:54] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [18:12:13] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [18:23:54] RECOVERY Free ram is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK: 20% free memory [18:25:56] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:28:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:29:26] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:29:46] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:35:35] PROBLEM host: wsexport.pmtpa.wmflabs is DOWN address: 10.4.0.231 CRITICAL - Host Unreachable (10.4.0.231) [18:36:55] PROBLEM host: ee-prototype.pmtpa.wmflabs is DOWN address: 10.4.0.93 CRITICAL - Host Unreachable (10.4.0.93) [18:42:14] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [18:45:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 240 processes [18:50:02] RECOVERY host: ee-prototype.pmtpa.wmflabs is UP address: 10.4.0.93 PING OK - Packet loss = 0%, RTA = 0.57 ms [18:50:25] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 102 processes [18:50:25] RECOVERY Current Load is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: OK - load average: 1.27, 0.45, 0.16 [18:51:05] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 118 processes [18:51:06] RECOVERY Disk Space is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: DISK OK [18:51:55] RECOVERY Free ram is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: OK: 72% free memory [18:53:39] RECOVERY SSH is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [18:57:15] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:57:45] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [18:58:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:59:35] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:02:44] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:02:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:05:35] PROBLEM host: wsexport.pmtpa.wmflabs is DOWN address: 10.4.0.231 CRITICAL - Host Unreachable (10.4.0.231) [19:08:25] PROBLEM Current Load is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: WARNING - load average: 19.05, 15.92, 9.44 [19:09:05] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 160 processes [19:12:23] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [19:18:24] PROBLEM Current Load is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CRITICAL - load average: 29.01, 25.14, 17.07 [19:24:04] PROBLEM Total processes is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS CRITICAL: 202 processes [19:27:34] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:29:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:29:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:33:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:34:54] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 13% free memory [19:35:54] PROBLEM host: wsexport.pmtpa.wmflabs is DOWN address: 10.4.0.231 CRITICAL - Host Unreachable (10.4.0.231) [19:42:34] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [19:44:54] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 2% free memory [19:51:06] RECOVERY Free ram is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: OK: 85% free memory [19:52:26] RECOVERY Total processes is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: PROCS OK: 105 processes [19:54:46] RECOVERY dpkg-check is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: All packages OK [19:55:25] RECOVERY Disk Space is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: DISK OK [19:55:45] RECOVERY Current Load is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: OK - load average: 1.17, 0.69, 0.56 [19:57:35] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:59:05] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:59:45] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:00:42] RECOVERY Free ram is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK: 94% free memory [20:00:42] RECOVERY dpkg-check is now: OK on wep2.pmtpa.wmflabs 10.4.0.57 output: All packages OK [20:00:42] RECOVERY Free ram is now: OK on authorlinks.pmtpa.wmflabs 10.4.1.62 output: OK: 79% free memory [20:00:42] RECOVERY Free ram is now: OK on bots-apache01.pmtpa.wmflabs 10.4.0.141 output: OK: 88% free memory [20:00:42] RECOVERY Free ram is now: OK on pediapress-ocg2.pmtpa.wmflabs 10.4.0.28 output: OK: 86% free memory [20:00:42] RECOVERY Total processes is now: OK on bastion2.pmtpa.wmflabs 10.4.1.84 output: PROCS OK: 104 processes [20:00:43] RECOVERY Total processes is now: OK on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: PROCS OK: 119 processes [20:00:43] RECOVERY Total processes is now: OK on fundraising-ci.pmtpa.wmflabs 10.4.0.221 output: PROCS OK: 97 processes [20:00:44] RECOVERY Current Load is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: OK - load average: 0.00, 0.04, 0.07 [20:00:44] RECOVERY Disk Space is now: OK on deployment-cache-mobile01.pmtpa.wmflabs 10.4.1.82 output: DISK OK [20:00:45] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [20:00:45] RECOVERY Disk Space is now: OK on piramido.pmtpa.wmflabs 10.4.1.16 output: DISK OK [20:00:46] RECOVERY Free ram is now: OK on ee-isarra.pmtpa.wmflabs 10.4.0.82 output: OK: 84% free memory [20:00:47] RECOVERY dpkg-check is now: OK on search1.pmtpa.wmflabs 10.4.0.100 output: All packages OK [20:00:47] RECOVERY Total processes is now: OK on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: PROCS OK: 94 processes [20:00:52] RECOVERY Disk Space is now: OK on signwriting-ase10.pmtpa.wmflabs 10.4.0.148 output: DISK OK [20:00:52] RECOVERY Disk Space is now: OK on stackfarm-web.pmtpa.wmflabs 10.4.0.113 output: DISK OK [20:00:52] RECOVERY Total processes is now: OK on tools-login.pmtpa.wmflabs 10.4.0.220 output: PROCS OK: 87 processes [20:00:52] RECOVERY Current Load is now: OK on mediawiki-dev-1.pmtpa.wmflabs 10.4.0.89 output: OK - load average: 0.00, 0.03, 0.05 [20:00:52] RECOVERY Current Load is now: OK on basictestsugar.pmtpa.wmflabs 10.4.1.10 output: OK - load average: 0.15, 0.05, 0.06 [20:00:53] RECOVERY Current Load is now: OK on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: OK - load average: 0.05, 0.04, 0.05 [20:00:53] RECOVERY Total processes is now: OK on npm-packaging.pmtpa.wmflabs 10.4.0.198 output: PROCS OK: 92 processes