[00:07:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [00:11:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [00:12:43] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [00:14:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:18:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [00:21:13] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:21:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:23:43] PROBLEM Disk Space is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK WARNING - free space: /tmp 0 MB (3% inode=99%): [00:33:42] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK CRITICAL - free space: /tmp 0 MB (2% inode=99%): [00:37:55] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [00:41:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [00:44:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:47:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 200 processes [00:48:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [00:52:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:52:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:02:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 215 processes [01:07:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 197 processes [01:09:24] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [01:09:44] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 291 processes [01:10:32] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [01:11:42] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [01:14:22] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [01:14:42] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 111 processes [01:15:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:18:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [01:22:24] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:23:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:28:24] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [01:40:35] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [01:42:35] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [01:46:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:48:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [01:52:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:53:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:10:53] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [02:12:43] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [02:17:25] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:18:35] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [02:22:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:24:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:40:55] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [02:42:45] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [02:47:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 212 processes [02:48:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:50:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [02:52:56] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:54:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:12:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [03:15:45] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [03:18:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:20:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [03:22:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:24:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:37:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 194 processes [03:42:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 203 processes [03:42:43] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [03:45:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [03:47:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 185 processes [03:49:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:52:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [03:54:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:13:02] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [04:14:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 391 processes [04:16:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [04:19:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:22:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [04:24:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:34:22] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 132 processes [04:42:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 275 processes [04:43:02] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [04:47:32] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [04:52:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:52:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [04:54:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:54:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:57:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 162 processes [05:13:02] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [05:17:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [05:23:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:23:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [05:24:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:25:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:44:43] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [05:47:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 214 processes [05:47:43] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [05:50:45] RECOVERY Total processes is now: OK on venus.pmtpa.wmflabs 10.4.0.66 output: PROCS OK: 105 processes [05:53:25] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [05:53:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:55:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:56:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:57:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 200 processes [06:15:34] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [06:15:44] RECOVERY Free ram is now: OK on swift-be1.pmtpa.wmflabs 10.4.0.107 output: OK: 33% free memory [06:17:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [06:23:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [06:24:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:25:46] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:27:26] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:27:46] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [06:35:24] RECOVERY Total processes is now: OK on php-packaging.pmtpa.wmflabs 10.4.0.177 output: PROCS OK: 105 processes [06:35:34] RECOVERY Free ram is now: OK on swift-aux1.pmtpa.wmflabs 10.4.0.154 output: OK: 85% free memory [06:35:34] RECOVERY Current Load is now: OK on nova-osm-keystone.pmtpa.wmflabs 10.4.0.130 output: OK - load average: 0.20, 0.17, 0.09 [06:35:44] RECOVERY Current Load is now: OK on apachemxetc.pmtpa.wmflabs 10.4.0.212 output: OK - load average: 0.00, 0.00, 0.00 [06:35:44] RECOVERY Total processes is now: OK on outreacheval.pmtpa.wmflabs 10.4.0.91 output: PROCS OK: 101 processes [06:35:44] RECOVERY Disk Space is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: DISK OK [06:35:44] RECOVERY Disk Space is now: OK on pediapress-ocg1.pmtpa.wmflabs 10.4.0.25 output: DISK OK [06:35:45] RECOVERY dpkg-check is now: OK on bots-dev.pmtpa.wmflabs 10.4.0.88 output: All packages OK [06:35:45] RECOVERY dpkg-check is now: OK on cn-wiki-db-lucid.pmtpa.wmflabs 10.4.0.151 output: All packages OK [06:35:45] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 0.80, 2.75, 2.62 [06:35:46] RECOVERY Total processes is now: OK on bots-dev.pmtpa.wmflabs 10.4.0.88 output: PROCS OK: 81 processes [06:35:46] RECOVERY Disk Space is now: OK on embed-sandbox.pmtpa.wmflabs 10.4.0.35 output: DISK OK [06:35:47] RECOVERY Current Load is now: OK on php-packaging.pmtpa.wmflabs 10.4.0.177 output: OK - load average: 0.01, 0.06, 0.02 [06:35:47] RECOVERY Current Load is now: OK on embed-sandbox.pmtpa.wmflabs 10.4.0.35 output: OK - load average: 0.10, 0.03, 0.01 [06:35:48] RECOVERY Disk Space is now: OK on firstinstance.pmtpa.wmflabs 10.4.0.95 output: DISK OK [06:35:48] RECOVERY Total processes is now: OK on search-test.pmtpa.wmflabs 10.4.0.12 output: PROCS OK: 94 processes [06:35:49] RECOVERY Free ram is now: OK on signwriting-ase11.pmtpa.wmflabs 10.4.0.15 output: OK: 75% free memory [06:35:49] RECOVERY Free ram is now: OK on staging.pmtpa.wmflabs 10.4.1.24 output: OK: 56% free memory [06:35:50] RECOVERY dpkg-check is now: OK on su-be2.pmtpa.wmflabs 10.4.0.183 output: All packages OK [06:35:54] RECOVERY Current Load is now: OK on bots-dev.pmtpa.wmflabs 10.4.0.88 output: OK - load average: 0.00, 0.00, 0.00 [06:35:54] RECOVERY Current Load is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK - load average: 0.25, 0.21, 0.24 [06:35:54] RECOVERY Disk Space is now: OK on build-precise1.pmtpa.wmflabs 10.4.0.173 output: DISK OK [06:35:54] RECOVERY Current Load is now: OK on centralauth-daemons.pmtpa.wmflabs 10.4.1.7 output: OK - load average: 0.00, 0.01, 0.05 [06:35:54] RECOVERY Current Load is now: OK on nginx-dev1.pmtpa.wmflabs 10.4.0.68 output: OK - load average: 0.00, 0.01, 0.00 [06:35:55] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [06:35:55] RECOVERY Free ram is now: OK on mingledbtest.pmtpa.wmflabs 10.4.0.131 output: OK: 91% free memory [06:35:56] RECOVERY Current Load is now: OK on otrs-jgreen.pmtpa.wmflabs 10.4.0.3 output: OK - load average: 0.00, 0.02, 0.00 [06:35:56] RECOVERY dpkg-check is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: All packages OK [06:36:04] RECOVERY Free ram is now: OK on robh2.pmtpa.wmflabs 10.4.0.96 output: OK: 84% free memory [06:36:04] RECOVERY Free ram is now: OK on robh2-mingle.pmtpa.wmflabs 10.4.0.134 output: OK: 80% free memory [06:36:04] RECOVERY Free ram is now: OK on scribunto.pmtpa.wmflabs 10.4.0.144 output: OK: 84% free memory [06:36:04] RECOVERY Disk Space is now: OK on search-test.pmtpa.wmflabs 10.4.0.12 output: DISK OK [06:36:04] RECOVERY dpkg-check is now: OK on signwriting-ase11.pmtpa.wmflabs 10.4.0.15 output: All packages OK [06:36:04] RECOVERY Total processes is now: OK on secondinstance.pmtpa.wmflabs 10.4.0.43 output: PROCS OK: 89 processes [06:36:05] RECOVERY Disk Space is now: OK on simplewikt.pmtpa.wmflabs 10.4.0.99 output: DISK OK [06:36:05] RECOVERY Current Load is now: OK on su-aux1.pmtpa.wmflabs 10.4.0.185 output: OK - load average: 0.00, 0.00, 0.00 [06:38:24] RECOVERY dpkg-check is now: OK on exim-test.pmtpa.wmflabs 10.4.0.168 output: All packages OK [06:38:24] PROBLEM Free ram is now: UNKNOWN on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: NRPE: Unable to read output [06:38:24] RECOVERY dpkg-check is now: OK on su-aux1.pmtpa.wmflabs 10.4.0.185 output: All packages OK [06:38:24] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 84% free memory [06:38:24] RECOVERY Current Load is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK - load average: 0.02, 0.02, 0.02 [06:38:25] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 30% free memory [06:38:25] PROBLEM Free ram is now: WARNING on aggregator1.pmtpa.wmflabs 10.4.0.79 output: Warning: 19% free memory [06:38:26] RECOVERY Current Load is now: OK on hugglewiki.pmtpa.wmflabs 10.4.0.50 output: OK - load average: 0.01, 0.06, 0.04 [06:38:26] RECOVERY Disk Space is now: OK on wikisource-web.pmtpa.wmflabs 10.4.0.78 output: DISK OK [06:38:27] RECOVERY Current Load is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK - load average: 0.00, 0.00, 0.00 [06:38:27] PROBLEM dpkg-check is now: CRITICAL on venus.pmtpa.wmflabs 10.4.0.66 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:38:28] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 29% free memory [06:38:28] RECOVERY Disk Space is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: DISK OK [06:38:34] RECOVERY Free ram is now: OK on su-be2.pmtpa.wmflabs 10.4.0.183 output: OK: 39% free memory [06:38:34] RECOVERY Current Load is now: OK on asher1.pmtpa.wmflabs 10.4.0.10 output: OK - load average: 0.02, 0.05, 0.01 [06:38:35] RECOVERY dpkg-check is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: All packages OK [06:38:35] RECOVERY Current Load is now: OK on cvresearch-web.pmtpa.wmflabs 10.4.1.18 output: OK - load average: 0.02, 0.11, 0.10 [06:38:35] RECOVERY Current Load is now: OK on bob.pmtpa.wmflabs 10.4.0.90 output: OK - load average: 0.00, 0.00, 0.00 [06:38:35] RECOVERY Current Load is now: OK on su-be3.pmtpa.wmflabs 10.4.0.191 output: OK - load average: 0.00, 0.00, 0.07 [06:38:35] RECOVERY Current Load is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK - load average: 0.20, 0.26, 0.32 [06:38:36] RECOVERY Total processes is now: OK on centralauth-daemons.pmtpa.wmflabs 10.4.1.7 output: PROCS OK: 89 processes [06:38:36] RECOVERY Disk Space is now: OK on cn-wiki-db-lucid.pmtpa.wmflabs 10.4.0.151 output: DISK OK [06:38:37] RECOVERY Current Load is now: OK on dev-solr.pmtpa.wmflabs 10.4.0.103 output: OK - load average: 0.00, 0.00, 0.00 [06:38:44] PROBLEM Total processes is now: CRITICAL on venus.pmtpa.wmflabs 10.4.0.66 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:38:44] RECOVERY Free ram is now: OK on integration-jenkins2.pmtpa.wmflabs 10.4.0.172 output: OK: 75% free memory [06:38:54] RECOVERY Disk Space is now: OK on robh2.pmtpa.wmflabs 10.4.0.96 output: DISK OK [06:38:54] RECOVERY Disk Space is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: DISK OK [06:38:54] RECOVERY Current Load is now: OK on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: OK - load average: 0.20, 0.29, 0.27 [06:38:54] RECOVERY Current Load is now: OK on gerrit-test.pmtpa.wmflabs 10.4.0.246 output: OK - load average: 0.20, 0.24, 0.23 [06:38:54] RECOVERY Current Load is now: OK on testing-yp.pmtpa.wmflabs 10.4.0.42 output: OK - load average: 0.87, 0.96, 0.63 [06:38:55] RECOVERY Total processes is now: OK on exim-test.pmtpa.wmflabs 10.4.0.168 output: PROCS OK: 88 processes [06:38:55] RECOVERY Disk Space is now: OK on outreacheval.pmtpa.wmflabs 10.4.0.91 output: DISK OK [06:38:56] RECOVERY Total processes is now: OK on wikibits-apache.pmtpa.wmflabs 10.4.0.209 output: PROCS OK: 103 processes [06:38:56] RECOVERY Current Load is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.09, 0.13, 0.17 [06:38:57] RECOVERY Total processes is now: OK on su-fe1.pmtpa.wmflabs 10.4.0.167 output: PROCS OK: 92 processes [06:38:57] RECOVERY Total processes is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: PROCS OK: 86 processes [06:38:58] RECOVERY Total processes is now: OK on signwriting-ase11.pmtpa.wmflabs 10.4.0.15 output: PROCS OK: 96 processes [06:39:04] RECOVERY Disk Space is now: OK on feeds.pmtpa.wmflabs 10.4.0.77 output: DISK OK [06:39:14] RECOVERY Total processes is now: OK on su-fe2.pmtpa.wmflabs 10.4.0.175 output: PROCS OK: 81 processes [06:39:14] RECOVERY dpkg-check is now: OK on centralauth-daemons.pmtpa.wmflabs 10.4.1.7 output: All packages OK [06:39:14] RECOVERY dpkg-check is now: OK on feeds.pmtpa.wmflabs 10.4.0.77 output: All packages OK [06:39:24] RECOVERY dpkg-check is now: OK on opengrok-web.pmtpa.wmflabs 10.4.0.133 output: All packages OK [06:39:24] RECOVERY dpkg-check is now: OK on pediapress-ocg1.pmtpa.wmflabs 10.4.0.25 output: All packages OK [06:39:24] RECOVERY Total processes is now: OK on hugglewa-db.pmtpa.wmflabs 10.4.0.108 output: PROCS OK: 87 processes [06:39:24] RECOVERY Disk Space is now: OK on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: DISK OK [06:39:24] RECOVERY Current Load is now: OK on integration-jenkins2.pmtpa.wmflabs 10.4.0.172 output: OK - load average: 0.08, 0.04, 0.05 [06:39:25] RECOVERY Disk Space is now: OK on mobile-b2g.pmtpa.wmflabs 10.4.0.241 output: DISK OK [06:39:25] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.00, 0.07, 0.07 [06:39:34] RECOVERY Current Load is now: OK on labs-relay.pmtpa.wmflabs 10.4.0.16 output: OK - load average: 0.00, 0.00, 0.00 [06:39:34] RECOVERY Total processes is now: OK on nagios-main.pmtpa.wmflabs 10.4.0.120 output: PROCS OK: 129 processes [06:39:34] RECOVERY dpkg-check is now: OK on publicdata-administration.pmtpa.wmflabs 10.4.0.101 output: All packages OK [06:39:34] RECOVERY Current Load is now: OK on mars.pmtpa.wmflabs 10.4.0.237 output: OK - load average: 0.02, 0.06, 0.05 [06:39:34] RECOVERY Disk Space is now: OK on signwriting-ase11.pmtpa.wmflabs 10.4.0.15 output: DISK OK [06:39:35] RECOVERY Disk Space is now: OK on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: DISK OK [06:39:35] RECOVERY Disk Space is now: OK on gerrit-test.pmtpa.wmflabs 10.4.0.246 output: DISK OK [06:39:36] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 79% free memory [06:39:36] RECOVERY Disk Space is now: OK on testing-yp.pmtpa.wmflabs 10.4.0.42 output: DISK OK [06:39:37] RECOVERY Free ram is now: OK on cvresearch-web.pmtpa.wmflabs 10.4.1.18 output: OK: 83% free memory [06:39:37] PROBLEM Free ram is now: UNKNOWN on gerrit-test.pmtpa.wmflabs 10.4.0.246 output: NRPE: Unable to read output [06:39:38] RECOVERY Total processes is now: OK on syslogcol-ab.pmtpa.wmflabs 10.4.0.223 output: PROCS OK: 103 processes [06:39:38] RECOVERY Total processes is now: OK on translatesvg.pmtpa.wmflabs 10.4.0.178 output: PROCS OK: 96 processes [06:39:39] RECOVERY Current Load is now: OK on feeds.pmtpa.wmflabs 10.4.0.77 output: OK - load average: 0.14, 0.05, 0.02 [06:39:39] RECOVERY Disk Space is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: DISK OK [06:39:40] RECOVERY dpkg-check is now: OK on wikibits-apache.pmtpa.wmflabs 10.4.0.209 output: All packages OK [06:39:40] RECOVERY Current Load is now: OK on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: OK - load average: 0.08, 0.09, 0.09 [06:39:41] RECOVERY dpkg-check is now: OK on master.pmtpa.wmflabs 10.4.0.14 output: All packages OK [06:39:44] RECOVERY Current Load is now: OK on incubator-sql.pmtpa.wmflabs 10.4.0.31 output: OK - load average: 0.58, 0.17, 0.10 [06:39:44] RECOVERY Free ram is now: OK on mediawikiglass.pmtpa.wmflabs 10.4.1.32 output: OK: 71% free memory [06:39:44] RECOVERY Current Load is now: OK on mobile-b2g.pmtpa.wmflabs 10.4.0.241 output: OK - load average: 0.40, 0.28, 0.14 [06:39:44] RECOVERY Free ram is now: OK on outreacheval.pmtpa.wmflabs 10.4.0.91 output: OK: 92% free memory [06:39:44] RECOVERY Current Load is now: OK on robh2-mingle.pmtpa.wmflabs 10.4.0.134 output: OK - load average: 0.01, 0.10, 0.13 [06:39:45] RECOVERY Free ram is now: OK on firstinstance.pmtpa.wmflabs 10.4.0.95 output: OK: 86% free memory [06:39:45] RECOVERY Total processes is now: OK on nova-osm-keystone.pmtpa.wmflabs 10.4.0.130 output: PROCS OK: 83 processes [06:39:46] RECOVERY dpkg-check is now: OK on su-fe1.pmtpa.wmflabs 10.4.0.167 output: All packages OK [06:39:46] RECOVERY Free ram is now: OK on syslogcol-ab.pmtpa.wmflabs 10.4.0.223 output: OK: 89% free memory [06:39:47] RECOVERY Free ram is now: OK on translatesvg.pmtpa.wmflabs 10.4.0.178 output: OK: 71% free memory [06:39:47] RECOVERY dpkg-check is now: OK on tutorial-mysql.pmtpa.wmflabs 10.4.0.170 output: All packages OK [06:39:48] RECOVERY Free ram is now: OK on wikibits-apache.pmtpa.wmflabs 10.4.0.209 output: OK: 83% free memory [06:39:54] RECOVERY Disk Space is now: OK on centralauth-daemons.pmtpa.wmflabs 10.4.1.7 output: DISK OK [06:39:54] RECOVERY Free ram is now: OK on incubator-sql.pmtpa.wmflabs 10.4.0.31 output: OK: 66% free memory [06:39:54] RECOVERY Total processes is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: PROCS OK: 80 processes [06:39:54] RECOVERY Free ram is now: OK on incubator-apache.pmtpa.wmflabs 10.4.0.116 output: OK: 85% free memory [06:39:54] RECOVERY Free ram is now: OK on wmde-test.pmtpa.wmflabs 10.4.0.186 output: OK: 85% free memory [06:39:55] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [06:39:55] RECOVERY Disk Space is now: OK on master.pmtpa.wmflabs 10.4.0.14 output: DISK OK [06:39:56] RECOVERY Current Load is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK - load average: 0.08, 0.18, 0.43 [06:39:56] RECOVERY dpkg-check is now: OK on nginx-ffuqua-doom1-3.pmtpa.wmflabs 10.4.0.80 output: All packages OK [06:40:14] PROBLEM Free ram is now: UNKNOWN on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: NRPE: Unable to read output [06:40:14] PROBLEM Free ram is now: UNKNOWN on testing-yp.pmtpa.wmflabs 10.4.0.42 output: NRPE: Unable to read output [06:40:24] PROBLEM Free ram is now: CRITICAL on venus.pmtpa.wmflabs 10.4.0.66 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:40:24] PROBLEM Disk Space is now: CRITICAL on venus.pmtpa.wmflabs 10.4.0.66 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:41:00] PROBLEM Current Load is now: CRITICAL on venus.pmtpa.wmflabs 10.4.0.66 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:42:24] RECOVERY dpkg-check is now: OK on testing-yp.pmtpa.wmflabs 10.4.0.42 output: All packages OK [06:45:34] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [06:47:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 191 processes [06:47:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [06:53:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [06:54:32] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:56:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:57:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:15:34] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [07:17:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [07:23:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [07:24:32] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:26:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:27:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:45:35] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [07:47:45] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [07:53:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [07:54:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:56:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:57:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:15:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [08:20:12] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [08:24:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:25:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [08:28:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:29:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:30:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 269 processes [08:45:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [08:50:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 139 processes [08:50:53] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [08:54:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:55:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [08:58:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:59:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:04:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 172 processes [09:15:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [09:21:22] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [09:25:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [09:26:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:28:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:29:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:45:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [09:51:24] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [09:52:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 203 processes [09:55:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [09:57:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:57:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 176 processes [09:58:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:59:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:12:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 203 processes [10:15:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [10:22:32] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [10:25:56] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [10:28:16] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:28:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:29:46] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:45:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [10:52:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [10:56:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [10:58:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:59:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:01:12] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:15:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [11:22:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [11:27:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [11:28:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:29:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:31:13] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:42:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 185 processes [11:45:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [11:52:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [11:57:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [11:58:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:59:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:01:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:05:34] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [12:17:13] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [12:22:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [12:27:45] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [12:28:55] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:29:05] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:32:22] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:32:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 203 processes [12:37:46] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 185 processes [12:47:15] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [12:52:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [12:52:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 217 processes [12:58:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [12:59:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:59:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:02:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:17:15] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [13:22:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [13:28:25] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [13:29:35] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:29:55] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:32:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:47:44] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [13:52:35] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [13:58:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [13:59:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:00:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:03:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:14:54] PROBLEM Free ram is now: UNKNOWN on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: NRPE: Call to fork() failed [14:15:45] PROBLEM Current Load is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:17:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [14:18:55] PROBLEM SSH is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Server answer: [14:19:55] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:20:22] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:20:32] PROBLEM Disk Space is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:22:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [14:23:52] RECOVERY SSH is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:24:52] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 22% free memory [14:25:24] RECOVERY dpkg-check is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: All packages OK [14:25:34] RECOVERY Disk Space is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: DISK OK [14:25:44] RECOVERY Current Load is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK - load average: 0.10, 0.77, 0.93 [14:28:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [14:29:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:33:13] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:33:23] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 248 processes [14:36:45] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.57, 9.76, 6.13 [14:36:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 13.77, 11.88, 7.39 [14:37:45] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 290 processes [14:37:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.66, 9.31, 6.54 [14:38:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 12.12, 10.90, 6.98 [14:42:22] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 10.99, 9.86, 6.98 [14:43:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 168 processes [14:43:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 10.52, 9.76, 6.64 [14:47:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [14:48:25] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 112 processes [14:55:55] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [14:58:41] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [14:59:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:00:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:04:13] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:17:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [15:25:56] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [15:28:36] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [15:29:46] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:31:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:34:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:47:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [15:55:56] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [15:58:36] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [15:58:46] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [15:59:46] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:01:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:04:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:17:55] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [16:26:14] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [16:30:52] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [16:31:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:34:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:34:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:37:46] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 200 processes [16:42:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 206 processes [16:50:32] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [16:56:14] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [17:00:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [17:02:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:04:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:04:58] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:06:45] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.21, 1.62, 4.19 [17:08:55] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 3.49, 3.16, 4.94 [17:12:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.86, 3.65, 4.76 [17:15:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [17:16:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 3.35, 3.56, 4.67 [17:20:33] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [17:20:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [17:26:14]