[19:58:33] RECOVERY host: bots-apache01.pmtpa.wmflabs is UP address: 10.4.0.141 PING OK - Packet loss = 0%, RTA = 0.73 ms [19:59:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:02:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [20:02:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:04:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:04:32] PROBLEM host: build2.pmtpa.wmflabs is DOWN address: 10.4.0.161 CRITICAL - Host Unreachable (10.4.0.161) [20:08:28] PROBLEM host: deployment-integration.pmtpa.wmflabs is DOWN address: 10.4.1.61 CRITICAL - Host Unreachable (10.4.1.61) [20:14:23] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [20:15:52] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [20:16:22] PROBLEM host: mwreview.pmtpa.wmflabs is DOWN address: 10.4.0.30 CRITICAL - Host Unreachable (10.4.0.30) [20:17:32] PROBLEM host: gadgets-apache3.pmtpa.wmflabs is DOWN address: 10.4.1.92 CRITICAL - Host Unreachable (10.4.1.92) [20:17:42] PROBLEM host: new-ns0.pmtpa.wmflabs is DOWN address: 10.4.0.190 CRITICAL - Host Unreachable (10.4.0.190) [20:18:02] PROBLEM host: incubator-mc.pmtpa.wmflabs is DOWN address: 10.4.0.20 CRITICAL - Host Unreachable (10.4.0.20) [20:18:02] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [20:18:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:18:42] PROBLEM host: maps-pgsql3.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [20:18:42] PROBLEM host: sf-book-server.pmtpa.wmflabs is DOWN address: 10.4.1.87 CRITICAL - Host Unreachable (10.4.1.87) [20:19:02] PROBLEM host: glam-gwtoolset-puppet.pmtpa.wmflabs is DOWN address: 10.4.0.34 CRITICAL - Host Unreachable (10.4.0.34) [20:19:02] PROBLEM host: mingle.pmtpa.wmflabs is DOWN address: 10.4.0.55 CRITICAL - Host Unreachable (10.4.0.55) [20:19:02] PROBLEM host: mobile-pg.pmtpa.wmflabs is DOWN address: 10.4.1.46 CRITICAL - Host Unreachable (10.4.1.46) [20:19:12] PROBLEM host: generator.pmtpa.wmflabs is DOWN address: 10.4.0.189 CRITICAL - Host Unreachable (10.4.0.189) [20:19:12] PROBLEM host: gerrit-dev-zap.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [20:19:12] PROBLEM host: home-migrate-precise.pmtpa.wmflabs is DOWN address: 10.4.1.45 CRITICAL - Host Unreachable (10.4.1.45) [20:19:12] PROBLEM host: integration-contintrefactor.pmtpa.wmflabs is DOWN address: 10.4.1.52 CRITICAL - Host Unreachable (10.4.1.52) [20:19:22] PROBLEM host: search1.pmtpa.wmflabs is DOWN address: 10.4.0.100 CRITICAL - Host Unreachable (10.4.0.100) [20:19:22] PROBLEM host: nginx-devunt.pmtpa.wmflabs is DOWN address: 10.4.0.65 CRITICAL - Host Unreachable (10.4.0.65) [20:19:22] PROBLEM host: npm-packaging.pmtpa.wmflabs is DOWN address: 10.4.0.198 CRITICAL - Host Unreachable (10.4.0.198) [20:19:22] PROBLEM host: bots-abogott-devel.pmtpa.wmflabs is DOWN address: 10.4.1.42 CRITICAL - Host Unreachable (10.4.1.42) [20:19:32] PROBLEM host: babbage0.pmtpa.wmflabs is DOWN address: 10.4.1.51 CRITICAL - Host Unreachable (10.4.1.51) [20:19:32] PROBLEM host: fundraising-ci.pmtpa.wmflabs is DOWN address: 10.4.0.221 CRITICAL - Host Unreachable (10.4.0.221) [20:19:32] PROBLEM host: redmine.pmtpa.wmflabs is DOWN address: 10.4.1.71 CRITICAL - Host Unreachable (10.4.1.71) [20:19:32] PROBLEM host: dumpmuncher.pmtpa.wmflabs is DOWN address: 10.4.0.32 CRITICAL - Host Unreachable (10.4.0.32) [20:19:32] PROBLEM host: bastion3.pmtpa.wmflabs is DOWN address: 10.4.1.85 CRITICAL - Host Unreachable (10.4.1.85) [20:19:32] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [20:19:42] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [20:19:42] PROBLEM host: dumps-bot3.pmtpa.wmflabs is DOWN address: 10.4.0.118 CRITICAL - Host Unreachable (10.4.0.118) [20:19:42] PROBLEM host: ve-roundtrip2.pmtpa.wmflabs is DOWN address: 10.4.0.162 CRITICAL - Host Unreachable (10.4.0.162) [20:19:42] PROBLEM host: gerrit-build.pmtpa.wmflabs is DOWN address: 10.4.0.70 CRITICAL - Host Unreachable (10.4.0.70) [20:19:42] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [20:19:42] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [20:19:42] PROBLEM host: lynwood.pmtpa.wmflabs is DOWN address: 10.4.0.2 CRITICAL - Host Unreachable (10.4.0.2) [20:19:52] PROBLEM host: nagios-dev.pmtpa.wmflabs is DOWN address: 10.4.0.201 CRITICAL - Host Unreachable (10.4.0.201) [20:19:52] PROBLEM host: bots-analytics.pmtpa.wmflabs is DOWN address: 10.4.1.44 CRITICAL - Host Unreachable (10.4.1.44) [20:19:52] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [20:21:33] PROBLEM host: checkwiki-web3.pmtpa.wmflabs is DOWN address: 10.4.1.74 CRITICAL - Host Unreachable (10.4.1.74) [20:21:43] PROBLEM host: node.pmtpa.wmflabs is DOWN address: 10.4.1.63 CRITICAL - Host Unreachable (10.4.1.63) [20:21:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:22:03] PROBLEM host: lucene-test1.pmtpa.wmflabs is DOWN address: 10.4.0.205 CRITICAL - Host Unreachable (10.4.0.205) [20:22:23] PROBLEM host: worker1.pmtpa.wmflabs is DOWN address: 10.4.1.12 CRITICAL - Host Unreachable (10.4.1.12) [20:22:43] PROBLEM host: authorlinks.pmtpa.wmflabs is DOWN address: 10.4.1.62 CRITICAL - Host Unreachable (10.4.1.62) [20:22:53] PROBLEM host: eventstream.pmtpa.wmflabs is DOWN address: 10.4.1.27 CRITICAL - Host Unreachable (10.4.1.27) [20:22:53] PROBLEM host: robh-mingle.pmtpa.wmflabs is DOWN address: 10.4.0.92 CRITICAL - Host Unreachable (10.4.0.92) [20:22:54] PROBLEM host: wep2.pmtpa.wmflabs is DOWN address: 10.4.0.57 CRITICAL - Host Unreachable (10.4.0.57) [20:23:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:23:13] PROBLEM host: webtools-login.pmtpa.wmflabs is DOWN address: 10.4.0.24 CRITICAL - Host Unreachable (10.4.0.24) [20:23:13] PROBLEM host: webtools-odie.pmtpa.wmflabs is DOWN address: 10.4.1.39 CRITICAL - Host Unreachable (10.4.1.39) [20:23:53] PROBLEM host: damian-keytest.pmtpa.wmflabs is DOWN address: 10.4.0.97 CRITICAL - Host Unreachable (10.4.0.97) [20:24:23] PROBLEM host: orgchart.pmtpa.wmflabs is DOWN address: 10.4.0.86 CRITICAL - Host Unreachable (10.4.0.86) [20:24:33] PROBLEM host: puppet1.pmtpa.wmflabs is DOWN address: 10.4.0.251 CRITICAL - Host Unreachable (10.4.0.251) [20:24:33] PROBLEM host: bots-nr2.pmtpa.wmflabs is DOWN address: 10.4.1.66 CRITICAL - Host Unreachable (10.4.1.66) [20:24:43] PROBLEM host: maps-osmmapnik.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [20:24:53] PROBLEM host: deployment-puptest.pmtpa.wmflabs is DOWN address: 10.4.0.195 CRITICAL - Host Unreachable (10.4.0.195) [20:29:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:34:33] PROBLEM host: build2.pmtpa.wmflabs is DOWN address: 10.4.0.161 CRITICAL - Host Unreachable (10.4.0.161) [20:37:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 6% free memory [20:38:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:39:13] PROBLEM host: deployment-integration.pmtpa.wmflabs is DOWN address: 10.4.1.61 CRITICAL - Host Unreachable (10.4.1.61) [20:39:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [20:39:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [20:39:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [20:40:22] PROBLEM host: openid-wiki.pmtpa.wmflabs is DOWN address: 10.4.1.50 PING CRITICAL - Packet loss = 100% [20:42:22] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 18% free memory [20:45:53] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [20:46:23] PROBLEM host: mwreview.pmtpa.wmflabs is DOWN address: 10.4.0.30 CRITICAL - Host Unreachable (10.4.0.30) [20:47:33] PROBLEM host: gadgets-apache3.pmtpa.wmflabs is DOWN address: 10.4.1.92 CRITICAL - Host Unreachable (10.4.1.92) [20:47:43] PROBLEM host: new-ns0.pmtpa.wmflabs is DOWN address: 10.4.0.190 CRITICAL - Host Unreachable (10.4.0.190) [20:47:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [20:48:03] PROBLEM host: incubator-mc.pmtpa.wmflabs is DOWN address: 10.4.0.20 CRITICAL - Host Unreachable (10.4.0.20) [20:48:03] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [20:48:45] PROBLEM host: maps-pgsql3.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [20:48:45] PROBLEM host: sf-book-server.pmtpa.wmflabs is DOWN address: 10.4.1.87 CRITICAL - Host Unreachable (10.4.1.87) [20:49:03] PROBLEM host: mingle.pmtpa.wmflabs is DOWN address: 10.4.0.55 CRITICAL - Host Unreachable (10.4.0.55) [20:49:04] PROBLEM host: mobile-pg.pmtpa.wmflabs is DOWN address: 10.4.1.46 CRITICAL - Host Unreachable (10.4.1.46) [20:49:13] PROBLEM host: generator.pmtpa.wmflabs is DOWN address: 10.4.0.189 CRITICAL - Host Unreachable (10.4.0.189) [20:49:13] PROBLEM host: gerrit-dev-zap.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [20:49:14] PROBLEM host: home-migrate-precise.pmtpa.wmflabs is DOWN address: 10.4.1.45 CRITICAL - Host Unreachable (10.4.1.45) [20:49:14] PROBLEM host: integration-contintrefactor.pmtpa.wmflabs is DOWN address: 10.4.1.52 CRITICAL - Host Unreachable (10.4.1.52) [20:49:23] PROBLEM host: search1.pmtpa.wmflabs is DOWN address: 10.4.0.100 CRITICAL - Host Unreachable (10.4.0.100) [20:49:23] PROBLEM host: nginx-devunt.pmtpa.wmflabs is DOWN address: 10.4.0.65 CRITICAL - Host Unreachable (10.4.0.65) [20:49:23] PROBLEM host: npm-packaging.pmtpa.wmflabs is DOWN address: 10.4.0.198 CRITICAL - Host Unreachable (10.4.0.198) [20:49:23] PROBLEM host: bots-abogott-devel.pmtpa.wmflabs is DOWN address: 10.4.1.42 CRITICAL - Host Unreachable (10.4.1.42) [20:49:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:49:33] PROBLEM host: fundraising-ci.pmtpa.wmflabs is DOWN address: 10.4.0.221 CRITICAL - Host Unreachable (10.4.0.221) [20:49:34] PROBLEM host: redmine.pmtpa.wmflabs is DOWN address: 10.4.1.71 CRITICAL - Host Unreachable (10.4.1.71) [20:49:34] PROBLEM host: dumpmuncher.pmtpa.wmflabs is DOWN address: 10.4.0.32 CRITICAL - Host Unreachable (10.4.0.32) [20:49:34] PROBLEM host: bastion3.pmtpa.wmflabs is DOWN address: 10.4.1.85 CRITICAL - Host Unreachable (10.4.1.85) [20:49:34] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [20:49:43] PROBLEM host: ve-roundtrip2.pmtpa.wmflabs is DOWN address: 10.4.0.162 CRITICAL - Host Unreachable (10.4.0.162) [20:49:43] PROBLEM host: gerrit-build.pmtpa.wmflabs is DOWN address: 10.4.0.70 CRITICAL - Host Unreachable (10.4.0.70) [20:49:44] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [20:49:44] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [20:49:44] PROBLEM host: glam-gwtoolset-puppet.pmtpa.wmflabs is DOWN address: 10.4.0.34 CRITICAL - Host Unreachable (10.4.0.34) [20:49:44] PROBLEM host: lynwood.pmtpa.wmflabs is DOWN address: 10.4.0.2 CRITICAL - Host Unreachable (10.4.0.2) [20:49:44] PROBLEM host: babbage0.pmtpa.wmflabs is DOWN address: 10.4.1.51 CRITICAL - Host Unreachable (10.4.1.51) [20:49:53] PROBLEM host: nagios-dev.pmtpa.wmflabs is DOWN address: 10.4.0.201 CRITICAL - Host Unreachable (10.4.0.201) [20:49:53] PROBLEM host: bots-analytics.pmtpa.wmflabs is DOWN address: 10.4.1.44 CRITICAL - Host Unreachable (10.4.1.44) [20:49:53] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [20:51:33] PROBLEM host: checkwiki-web3.pmtpa.wmflabs is DOWN address: 10.4.1.74 CRITICAL - Host Unreachable (10.4.1.74) [20:51:43] PROBLEM host: node.pmtpa.wmflabs is DOWN address: 10.4.1.63 CRITICAL - Host Unreachable (10.4.1.63) [20:51:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:52:03] PROBLEM host: lucene-test1.pmtpa.wmflabs is DOWN address: 10.4.0.205 CRITICAL - Host Unreachable (10.4.0.205) [20:52:23] PROBLEM host: worker1.pmtpa.wmflabs is DOWN address: 10.4.1.12 CRITICAL - Host Unreachable (10.4.1.12) [20:52:33] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [20:52:33] PROBLEM host: dumps-bot3.pmtpa.wmflabs is DOWN address: 10.4.0.118 CRITICAL - Host Unreachable (10.4.0.118) [20:52:43] PROBLEM host: authorlinks.pmtpa.wmflabs is DOWN address: 10.4.1.62 CRITICAL - Host Unreachable (10.4.1.62) [20:52:53] PROBLEM host: robh-mingle.pmtpa.wmflabs is DOWN address: 10.4.0.92 CRITICAL - Host Unreachable (10.4.0.92) [20:52:53] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 4% free memory [20:52:53] PROBLEM host: wep2.pmtpa.wmflabs is DOWN address: 10.4.0.57 CRITICAL - Host Unreachable (10.4.0.57) [20:53:13] PROBLEM host: eventstream.pmtpa.wmflabs is DOWN address: 10.4.1.27 CRITICAL - Host Unreachable (10.4.1.27) [20:53:13] PROBLEM host: webtools-login.pmtpa.wmflabs is DOWN address: 10.4.0.24 CRITICAL - Host Unreachable (10.4.0.24) [20:53:13] PROBLEM host: webtools-odie.pmtpa.wmflabs is DOWN address: 10.4.1.39 CRITICAL - Host Unreachable (10.4.1.39) [20:53:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:53:53] PROBLEM host: damian-keytest.pmtpa.wmflabs is DOWN address: 10.4.0.97 CRITICAL - Host Unreachable (10.4.0.97) [20:54:23] PROBLEM host: orgchart.pmtpa.wmflabs is DOWN address: 10.4.0.86 CRITICAL - Host Unreachable (10.4.0.86) [20:54:33] PROBLEM host: puppet1.pmtpa.wmflabs is DOWN address: 10.4.0.251 CRITICAL - Host Unreachable (10.4.0.251) [20:54:33] PROBLEM host: bots-nr2.pmtpa.wmflabs is DOWN address: 10.4.1.66 CRITICAL - Host Unreachable (10.4.1.66) [20:54:43] PROBLEM host: maps-osmmapnik.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [20:54:53] PROBLEM host: deployment-puptest.pmtpa.wmflabs is DOWN address: 10.4.0.195 CRITICAL - Host Unreachable (10.4.0.195) [21:01:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [21:02:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:05:33] PROBLEM host: build2.pmtpa.wmflabs is DOWN address: 10.4.0.161 CRITICAL - Host Unreachable (10.4.0.161) [21:07:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [21:09:13] PROBLEM host: deployment-integration.pmtpa.wmflabs is DOWN address: 10.4.1.61 CRITICAL - Host Unreachable (10.4.1.61) [21:10:53] PROBLEM host: openid-wiki.pmtpa.wmflabs is DOWN address: 10.4.1.50 CRITICAL - Host Unreachable (10.4.1.50) [21:12:33] RECOVERY host: openid-wiki.pmtpa.wmflabs is UP address: 10.4.1.50 PING OK - Packet loss = 0%, RTA = 0.96 ms [21:16:12] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [21:18:02] PROBLEM host: gadgets-apache3.pmtpa.wmflabs is DOWN address: 10.4.1.92 CRITICAL - Host Unreachable (10.4.1.92) [21:18:32] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [21:18:42] PROBLEM host: incubator-mc.pmtpa.wmflabs is DOWN address: 10.4.0.20 CRITICAL - Host Unreachable (10.4.0.20) [21:19:02] PROBLEM host: maps-pgsql3.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [21:19:22] PROBLEM host: mingle.pmtpa.wmflabs is DOWN address: 10.4.0.55 CRITICAL - Host Unreachable (10.4.0.55) [21:19:22] PROBLEM host: mwreview.pmtpa.wmflabs is DOWN address: 10.4.0.30 CRITICAL - Host Unreachable (10.4.0.30) [21:19:22] PROBLEM host: new-ns0.pmtpa.wmflabs is DOWN address: 10.4.0.190 CRITICAL - Host Unreachable (10.4.0.190) [21:19:32] PROBLEM host: sf-book-server.pmtpa.wmflabs is DOWN address: 10.4.1.87 CRITICAL - Host Unreachable (10.4.1.87) [21:19:42] PROBLEM host: dumpmuncher.pmtpa.wmflabs is DOWN address: 10.4.0.32 CRITICAL - Host Unreachable (10.4.0.32) [21:19:42] PROBLEM host: fundraising-ci.pmtpa.wmflabs is DOWN address: 10.4.0.221 CRITICAL - Host Unreachable (10.4.0.221) [21:19:43] PROBLEM host: generator.pmtpa.wmflabs is DOWN address: 10.4.0.189 CRITICAL - Host Unreachable (10.4.0.189) [21:19:43] PROBLEM host: home-migrate-precise.pmtpa.wmflabs is DOWN address: 10.4.1.45 CRITICAL - Host Unreachable (10.4.1.45) [21:19:43] PROBLEM host: integration-contintrefactor.pmtpa.wmflabs is DOWN address: 10.4.1.52 CRITICAL - Host Unreachable (10.4.1.52) [21:19:43] PROBLEM host: gerrit-dev-zap.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [21:19:43] PROBLEM host: mobile-pg.pmtpa.wmflabs is DOWN address: 10.4.1.46 CRITICAL - Host Unreachable (10.4.1.46) [21:19:52] PROBLEM host: nginx-devunt.pmtpa.wmflabs is DOWN address: 10.4.0.65 CRITICAL - Host Unreachable (10.4.0.65) [21:19:52] PROBLEM host: search1.pmtpa.wmflabs is DOWN address: 10.4.0.100 CRITICAL - Host Unreachable (10.4.0.100) [21:19:52] PROBLEM host: npm-packaging.pmtpa.wmflabs is DOWN address: 10.4.0.198 CRITICAL - Host Unreachable (10.4.0.198) [21:19:52] PROBLEM host: babbage0.pmtpa.wmflabs is DOWN address: 10.4.1.51 CRITICAL - Host Unreachable (10.4.1.51) [21:19:52] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [21:19:52] PROBLEM host: lynwood.pmtpa.wmflabs is DOWN address: 10.4.0.2 CRITICAL - Host Unreachable (10.4.0.2) [21:19:52] PROBLEM host: ve-roundtrip2.pmtpa.wmflabs is DOWN address: 10.4.0.162 CRITICAL - Host Unreachable (10.4.0.162) [21:20:43] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [21:20:53] PROBLEM host: gerrit-build.pmtpa.wmflabs is DOWN address: 10.4.0.70 CRITICAL - Host Unreachable (10.4.0.70) [21:20:53] PROBLEM host: glam-gwtoolset-puppet.pmtpa.wmflabs is DOWN address: 10.4.0.34 CRITICAL - Host Unreachable (10.4.0.34) [21:20:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:21:23] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [21:21:39] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [21:21:39] PROBLEM host: checkwiki-web3.pmtpa.wmflabs is DOWN address: 10.4.1.74 CRITICAL - Host Unreachable (10.4.1.74) [21:21:43] PROBLEM host: node.pmtpa.wmflabs is DOWN address: 10.4.1.63 CRITICAL - Host Unreachable (10.4.1.63) [21:21:43] PROBLEM host: bastion3.pmtpa.wmflabs is DOWN address: 10.4.1.85 CRITICAL - Host Unreachable (10.4.1.85) [21:21:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:22:03] PROBLEM host: lucene-test1.pmtpa.wmflabs is DOWN address: 10.4.0.205 CRITICAL - Host Unreachable (10.4.0.205) [21:22:23] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [21:22:23] PROBLEM host: worker1.pmtpa.wmflabs is DOWN address: 10.4.1.12 CRITICAL - Host Unreachable (10.4.1.12) [21:22:33] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [21:22:33] PROBLEM host: dumps-bot3.pmtpa.wmflabs is DOWN address: 10.4.0.118 CRITICAL - Host Unreachable (10.4.0.118) [21:22:43] PROBLEM host: nagios-dev.pmtpa.wmflabs is DOWN address: 10.4.0.201 CRITICAL - Host Unreachable (10.4.0.201) [21:22:43] PROBLEM host: authorlinks.pmtpa.wmflabs is DOWN address: 10.4.1.62 CRITICAL - Host Unreachable (10.4.1.62) [21:22:53] PROBLEM host: redmine.pmtpa.wmflabs is DOWN address: 10.4.1.71 CRITICAL - Host Unreachable (10.4.1.71) [21:22:53] PROBLEM host: robh-mingle.pmtpa.wmflabs is DOWN address: 10.4.0.92 CRITICAL - Host Unreachable (10.4.0.92) [21:22:53] PROBLEM host: wep2.pmtpa.wmflabs is DOWN address: 10.4.0.57 CRITICAL - Host Unreachable (10.4.0.57) [21:23:03] PROBLEM host: bots-abogott-devel.pmtpa.wmflabs is DOWN address: 10.4.1.42 CRITICAL - Host Unreachable (10.4.1.42) [21:23:03] PROBLEM host: bots-analytics.pmtpa.wmflabs is DOWN address: 10.4.1.44 CRITICAL - Host Unreachable (10.4.1.44) [21:23:13] PROBLEM host: eventstream.pmtpa.wmflabs is DOWN address: 10.4.1.27 CRITICAL - Host Unreachable (10.4.1.27) [21:23:13] PROBLEM host: webtools-login.pmtpa.wmflabs is DOWN address: 10.4.0.24 CRITICAL - Host Unreachable (10.4.0.24) [21:23:13] PROBLEM host: webtools-odie.pmtpa.wmflabs is DOWN address: 10.4.1.39 CRITICAL - Host Unreachable (10.4.1.39) [21:23:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:23:53] PROBLEM host: damian-keytest.pmtpa.wmflabs is DOWN address: 10.4.0.97 CRITICAL - Host Unreachable (10.4.0.97) [21:24:23] PROBLEM host: orgchart.pmtpa.wmflabs is DOWN address: 10.4.0.86 CRITICAL - Host Unreachable (10.4.0.86) [21:24:33] PROBLEM host: puppet1.pmtpa.wmflabs is DOWN address: 10.4.0.251 CRITICAL - Host Unreachable (10.4.0.251) [21:24:33] PROBLEM host: bots-nr2.pmtpa.wmflabs is DOWN address: 10.4.1.66 CRITICAL - Host Unreachable (10.4.1.66) [21:24:43] PROBLEM host: maps-osmmapnik.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [21:24:53] PROBLEM host: deployment-puptest.pmtpa.wmflabs is DOWN address: 10.4.0.195 CRITICAL - Host Unreachable (10.4.0.195) [21:29:22] RECOVERY host: orgchart.pmtpa.wmflabs is UP address: 10.4.0.86 PING OK - Packet loss = 0%, RTA = 0.99 ms [21:32:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:35:33] PROBLEM host: build2.pmtpa.wmflabs is DOWN address: 10.4.0.161 CRITICAL - Host Unreachable (10.4.0.161) [21:39:13] PROBLEM host: deployment-integration.pmtpa.wmflabs is DOWN address: 10.4.1.61 CRITICAL - Host Unreachable (10.4.1.61) [21:46:13] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [21:48:03] PROBLEM host: gadgets-apache3.pmtpa.wmflabs is DOWN address: 10.4.1.92 CRITICAL - Host Unreachable (10.4.1.92) [21:48:43] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [21:49:05] PROBLEM host: incubator-mc.pmtpa.wmflabs is DOWN address: 10.4.0.20 CRITICAL - Host Unreachable (10.4.0.20) [21:49:05] PROBLEM host: maps-pgsql3.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [21:49:23] PROBLEM host: mingle.pmtpa.wmflabs is DOWN address: 10.4.0.55 CRITICAL - Host Unreachable (10.4.0.55) [21:49:23] PROBLEM host: mwreview.pmtpa.wmflabs is DOWN address: 10.4.0.30 CRITICAL - Host Unreachable (10.4.0.30) [21:49:23] PROBLEM host: new-ns0.pmtpa.wmflabs is DOWN address: 10.4.0.190 CRITICAL - Host Unreachable (10.4.0.190) [21:49:33] PROBLEM host: sf-book-server.pmtpa.wmflabs is DOWN address: 10.4.1.87 CRITICAL - Host Unreachable (10.4.1.87) [21:49:43] PROBLEM host: fundraising-ci.pmtpa.wmflabs is DOWN address: 10.4.0.221 CRITICAL - Host Unreachable (10.4.0.221) [21:49:43] PROBLEM host: generator.pmtpa.wmflabs is DOWN address: 10.4.0.189 CRITICAL - Host Unreachable (10.4.0.189) [21:49:43] PROBLEM host: home-migrate-precise.pmtpa.wmflabs is DOWN address: 10.4.1.45 CRITICAL - Host Unreachable (10.4.1.45) [21:49:43] PROBLEM host: integration-contintrefactor.pmtpa.wmflabs is DOWN address: 10.4.1.52 CRITICAL - Host Unreachable (10.4.1.52) [21:49:43] PROBLEM host: gerrit-dev-zap.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [21:49:44] PROBLEM host: mobile-pg.pmtpa.wmflabs is DOWN address: 10.4.1.46 CRITICAL - Host Unreachable (10.4.1.46) [21:49:53] PROBLEM host: nginx-devunt.pmtpa.wmflabs is DOWN address: 10.4.0.65 CRITICAL - Host Unreachable (10.4.0.65) [21:49:53] PROBLEM host: search1.pmtpa.wmflabs is DOWN address: 10.4.0.100 CRITICAL - Host Unreachable (10.4.0.100) [21:49:53] PROBLEM host: npm-packaging.pmtpa.wmflabs is DOWN address: 10.4.0.198 CRITICAL - Host Unreachable (10.4.0.198) [21:49:53] PROBLEM host: babbage0.pmtpa.wmflabs is DOWN address: 10.4.1.51 CRITICAL - Host Unreachable (10.4.1.51) [21:49:53] PROBLEM host: ve-roundtrip2.pmtpa.wmflabs is DOWN address: 10.4.0.162 CRITICAL - Host Unreachable (10.4.0.162) [21:49:53] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [21:50:13] PROBLEM host: dumpmuncher.pmtpa.wmflabs is DOWN address: 10.4.0.32 CRITICAL - Host Unreachable (10.4.0.32) [21:50:43] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [21:50:53] PROBLEM host: glam-gwtoolset-puppet.pmtpa.wmflabs is DOWN address: 10.4.0.34 CRITICAL - Host Unreachable (10.4.0.34) [21:50:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:51:03] PROBLEM host: lynwood.pmtpa.wmflabs is DOWN address: 10.4.0.2 CRITICAL - Host Unreachable (10.4.0.2) [21:51:23] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [21:51:33] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [21:51:33] PROBLEM host: checkwiki-web3.pmtpa.wmflabs is DOWN address: 10.4.1.74 CRITICAL - Host Unreachable (10.4.1.74) [21:51:43] PROBLEM host: node.pmtpa.wmflabs is DOWN address: 10.4.1.63 CRITICAL - Host Unreachable (10.4.1.63) [21:51:43] PROBLEM host: bastion3.pmtpa.wmflabs is DOWN address: 10.4.1.85 CRITICAL - Host Unreachable (10.4.1.85) [21:51:53] PROBLEM host: gerrit-build.pmtpa.wmflabs is DOWN address: 10.4.0.70 CRITICAL - Host Unreachable (10.4.0.70) [21:51:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:52:03] PROBLEM host: lucene-test1.pmtpa.wmflabs is DOWN address: 10.4.0.205 CRITICAL - Host Unreachable (10.4.0.205) [21:52:23] PROBLEM host: worker1.pmtpa.wmflabs is DOWN address: 10.4.1.12 CRITICAL - Host Unreachable (10.4.1.12) [21:52:33] PROBLEM host: dumps-bot3.pmtpa.wmflabs is DOWN address: 10.4.0.118 CRITICAL - Host Unreachable (10.4.0.118) [21:52:43] PROBLEM host: nagios-dev.pmtpa.wmflabs is DOWN address: 10.4.0.201 CRITICAL - Host Unreachable (10.4.0.201) [21:52:43] PROBLEM host: authorlinks.pmtpa.wmflabs is DOWN address: 10.4.1.62 CRITICAL - Host Unreachable (10.4.1.62) [21:52:43] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [21:52:53] PROBLEM host: redmine.pmtpa.wmflabs is DOWN address: 10.4.1.71 CRITICAL - Host Unreachable (10.4.1.71) [21:52:53] PROBLEM host: robh-mingle.pmtpa.wmflabs is DOWN address: 10.4.0.92 CRITICAL - Host Unreachable (10.4.0.92) [21:52:53] PROBLEM host: wep2.pmtpa.wmflabs is DOWN address: 10.4.0.57 CRITICAL - Host Unreachable (10.4.0.57) [21:53:03] PROBLEM host: bots-abogott-devel.pmtpa.wmflabs is DOWN address: 10.4.1.42 CRITICAL - Host Unreachable (10.4.1.42) [21:53:03] PROBLEM host: bots-analytics.pmtpa.wmflabs is DOWN address: 10.4.1.44 CRITICAL - Host Unreachable (10.4.1.44) [21:53:13] PROBLEM host: eventstream.pmtpa.wmflabs is DOWN address: 10.4.1.27 CRITICAL - Host Unreachable (10.4.1.27) [21:53:13] PROBLEM host: webtools-login.pmtpa.wmflabs is DOWN address: 10.4.0.24 CRITICAL - Host Unreachable (10.4.0.24) [21:53:13] PROBLEM host: webtools-odie.pmtpa.wmflabs is DOWN address: 10.4.1.39 CRITICAL - Host Unreachable (10.4.1.39) [21:53:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:53:53] PROBLEM host: damian-keytest.pmtpa.wmflabs is DOWN address: 10.4.0.97 CRITICAL - Host Unreachable (10.4.0.97) [21:54:34] PROBLEM host: puppet1.pmtpa.wmflabs is DOWN address: 10.4.0.251 CRITICAL - Host Unreachable (10.4.0.251) [21:54:34] PROBLEM host: bots-nr2.pmtpa.wmflabs is DOWN address: 10.4.1.66 CRITICAL - Host Unreachable (10.4.1.66) [21:54:43] PROBLEM host: maps-osmmapnik.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [21:54:53] PROBLEM host: deployment-puptest.pmtpa.wmflabs is DOWN address: 10.4.0.195 CRITICAL - Host Unreachable (10.4.0.195) [22:02:53] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:05:33] PROBLEM host: build2.pmtpa.wmflabs is DOWN address: 10.4.0.161 CRITICAL - Host Unreachable (10.4.0.161) [22:07:23] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [22:09:13] PROBLEM host: deployment-integration.pmtpa.wmflabs is DOWN address: 10.4.1.61 CRITICAL - Host Unreachable (10.4.1.61) [22:16:13] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [22:17:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [22:18:03] PROBLEM host: gadgets-apache3.pmtpa.wmflabs is DOWN address: 10.4.1.92 CRITICAL - Host Unreachable (10.4.1.92) [22:18:43] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [22:19:03] PROBLEM host: maps-pgsql3.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [22:19:13] PROBLEM host: incubator-mc.pmtpa.wmflabs is DOWN address: 10.4.0.20 CRITICAL - Host Unreachable (10.4.0.20) [22:19:23] PROBLEM host: mingle.pmtpa.wmflabs is DOWN address: 10.4.0.55 CRITICAL - Host Unreachable (10.4.0.55) [22:19:23] PROBLEM host: mwreview.pmtpa.wmflabs is DOWN address: 10.4.0.30 CRITICAL - Host Unreachable (10.4.0.30) [22:19:23] PROBLEM host: new-ns0.pmtpa.wmflabs is DOWN address: 10.4.0.190 CRITICAL - Host Unreachable (10.4.0.190) [22:19:33] PROBLEM host: sf-book-server.pmtpa.wmflabs is DOWN address: 10.4.1.87 CRITICAL - Host Unreachable (10.4.1.87) [22:19:43] PROBLEM host: fundraising-ci.pmtpa.wmflabs is DOWN address: 10.4.0.221 CRITICAL - Host Unreachable (10.4.0.221) [22:19:43] PROBLEM host: generator.pmtpa.wmflabs is DOWN address: 10.4.0.189 CRITICAL - Host Unreachable (10.4.0.189) [22:19:43] PROBLEM host: home-migrate-precise.pmtpa.wmflabs is DOWN address: 10.4.1.45 CRITICAL - Host Unreachable (10.4.1.45) [22:19:43] PROBLEM host: integration-contintrefactor.pmtpa.wmflabs is DOWN address: 10.4.1.52 CRITICAL - Host Unreachable (10.4.1.52) [22:19:43] PROBLEM host: gerrit-dev-zap.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [22:19:43] PROBLEM host: mobile-pg.pmtpa.wmflabs is DOWN address: 10.4.1.46 CRITICAL - Host Unreachable (10.4.1.46) [22:19:53] PROBLEM host: nginx-devunt.pmtpa.wmflabs is DOWN address: 10.4.0.65 CRITICAL - Host Unreachable (10.4.0.65) [22:19:53] PROBLEM host: search1.pmtpa.wmflabs is DOWN address: 10.4.0.100 CRITICAL - Host Unreachable (10.4.0.100) [22:19:53] PROBLEM host: npm-packaging.pmtpa.wmflabs is DOWN address: 10.4.0.198 CRITICAL - Host Unreachable (10.4.0.198) [22:19:53] PROBLEM host: babbage0.pmtpa.wmflabs is DOWN address: 10.4.1.51 CRITICAL - Host Unreachable (10.4.1.51) [22:19:53] PROBLEM host: ve-roundtrip2.pmtpa.wmflabs is DOWN address: 10.4.0.162 CRITICAL - Host Unreachable (10.4.0.162) [22:19:53] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [22:20:52] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [22:21:12] PROBLEM host: dumpmuncher.pmtpa.wmflabs is DOWN address: 10.4.0.32 CRITICAL - Host Unreachable (10.4.0.32) [22:21:42] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [22:21:42] PROBLEM host: glam-gwtoolset-puppet.pmtpa.wmflabs is DOWN address: 10.4.0.34 CRITICAL - Host Unreachable (10.4.0.34) [22:21:52] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [22:22:02] PROBLEM host: lynwood.pmtpa.wmflabs is DOWN address: 10.4.0.2 CRITICAL - Host Unreachable (10.4.0.2) [22:22:02] PROBLEM host: bastion3.pmtpa.wmflabs is DOWN address: 10.4.1.85 CRITICAL - Host Unreachable (10.4.1.85) [22:22:13] PROBLEM host: lucene-test1.pmtpa.wmflabs is DOWN address: 10.4.0.205 CRITICAL - Host Unreachable (10.4.0.205) [22:22:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:22:32] PROBLEM host: worker1.pmtpa.wmflabs is DOWN address: 10.4.1.12 CRITICAL - Host Unreachable (10.4.1.12) [22:22:52] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [22:22:52] PROBLEM host: dumps-bot3.pmtpa.wmflabs is DOWN address: 10.4.0.118 CRITICAL - Host Unreachable (10.4.0.118) [22:22:52] PROBLEM host: nagios-dev.pmtpa.wmflabs is DOWN address: 10.4.0.201 CRITICAL - Host Unreachable (10.4.0.201) [22:22:52] PROBLEM host: gerrit-build.pmtpa.wmflabs is DOWN address: 10.4.0.70 CRITICAL - Host Unreachable (10.4.0.70) [22:23:02] PROBLEM host: wep2.pmtpa.wmflabs is DOWN address: 10.4.0.57 CRITICAL - Host Unreachable (10.4.0.57) [22:23:02] PROBLEM host: node.pmtpa.wmflabs is DOWN address: 10.4.1.63 CRITICAL - Host Unreachable (10.4.1.63) [22:23:02] PROBLEM host: redmine.pmtpa.wmflabs is DOWN address: 10.4.1.71 CRITICAL - Host Unreachable (10.4.1.71) [22:23:22] PROBLEM host: authorlinks.pmtpa.wmflabs is DOWN address: 10.4.1.62 CRITICAL - Host Unreachable (10.4.1.62) [22:23:22] PROBLEM host: bots-abogott-devel.pmtpa.wmflabs is DOWN address: 10.4.1.42 CRITICAL - Host Unreachable (10.4.1.42) [22:23:32] PROBLEM host: eventstream.pmtpa.wmflabs is DOWN address: 10.4.1.27 CRITICAL - Host Unreachable (10.4.1.27) [22:23:42] PROBLEM host: robh-mingle.pmtpa.wmflabs is DOWN address: 10.4.0.92 CRITICAL - Host Unreachable (10.4.0.92) [22:23:42] PROBLEM host: checkwiki-web3.pmtpa.wmflabs is DOWN address: 10.4.1.74 CRITICAL - Host Unreachable (10.4.1.74) [22:23:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:23:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:24:12] PROBLEM host: bots-analytics.pmtpa.wmflabs is DOWN address: 10.4.1.44 CRITICAL - Host Unreachable (10.4.1.44) [22:24:22] PROBLEM host: webtools-login.pmtpa.wmflabs is DOWN address: 10.4.0.24 CRITICAL - Host Unreachable (10.4.0.24) [22:24:22] PROBLEM host: webtools-odie.pmtpa.wmflabs is DOWN address: 10.4.1.39 CRITICAL - Host Unreachable (10.4.1.39) [22:24:32] PROBLEM host: damian-keytest.pmtpa.wmflabs is DOWN address: 10.4.0.97 CRITICAL - Host Unreachable (10.4.0.97) [22:24:52] PROBLEM host: bots-nr2.pmtpa.wmflabs is DOWN address: 10.4.1.66 CRITICAL - Host Unreachable (10.4.1.66) [22:25:02] PROBLEM host: deployment-puptest.pmtpa.wmflabs is DOWN address: 10.4.0.195 CRITICAL - Host Unreachable (10.4.0.195) [22:25:23] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [22:27:03] PROBLEM host: maps-osmmapnik.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [22:27:29] PROBLEM host: puppet1.pmtpa.wmflabs is DOWN address: 10.4.0.251 CRITICAL - Host Unreachable (10.4.0.251) [22:33:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:35:33] PROBLEM host: build2.pmtpa.wmflabs is DOWN address: 10.4.0.161 CRITICAL - Host Unreachable (10.4.0.161) [22:37:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 7% free memory [22:39:13] PROBLEM host: deployment-integration.pmtpa.wmflabs is DOWN address: 10.4.1.61 CRITICAL - Host Unreachable (10.4.1.61) [22:42:23] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [22:46:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [22:48:42] PROBLEM host: gadgets-apache3.pmtpa.wmflabs is DOWN address: 10.4.1.92 CRITICAL - Host Unreachable (10.4.1.92) [22:49:12] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [22:49:26] PROBLEM host: maps-pgsql3.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [22:49:33] PROBLEM host: incubator-mc.pmtpa.wmflabs is DOWN address: 10.4.0.20 CRITICAL - Host Unreachable (10.4.0.20) [22:49:42] PROBLEM host: mingle.pmtpa.wmflabs is DOWN address: 10.4.0.55 CRITICAL - Host Unreachable (10.4.0.55) [22:49:52] PROBLEM host: mwreview.pmtpa.wmflabs is DOWN address: 10.4.0.30 CRITICAL - Host Unreachable (10.4.0.30) [22:49:52] PROBLEM host: new-ns0.pmtpa.wmflabs is DOWN address: 10.4.0.190 CRITICAL - Host Unreachable (10.4.0.190) [22:49:52] PROBLEM host: mobile-pg.pmtpa.wmflabs is DOWN address: 10.4.1.46 CRITICAL - Host Unreachable (10.4.1.46) [22:49:52] PROBLEM host: generator.pmtpa.wmflabs is DOWN address: 10.4.0.189 CRITICAL - Host Unreachable (10.4.0.189) [22:50:22] PROBLEM host: fundraising-ci.pmtpa.wmflabs is DOWN address: 10.4.0.221 CRITICAL - Host Unreachable (10.4.0.221) [22:50:32] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [22:50:32] PROBLEM host: integration-contintrefactor.pmtpa.wmflabs is DOWN address: 10.4.1.52 CRITICAL - Host Unreachable (10.4.1.52) [22:50:52] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [22:51:22] PROBLEM host: gerrit-dev-zap.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [22:51:22] PROBLEM host: nginx-devunt.pmtpa.wmflabs is DOWN address: 10.4.0.65 CRITICAL - Host Unreachable (10.4.0.65) [22:51:32] PROBLEM host: home-migrate-precise.pmtpa.wmflabs is DOWN address: 10.4.1.45 CRITICAL - Host Unreachable (10.4.1.45) [22:51:32] PROBLEM host: search1.pmtpa.wmflabs is DOWN address: 10.4.0.100 CRITICAL - Host Unreachable (10.4.0.100) [22:51:42] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [22:51:42] PROBLEM host: glam-gwtoolset-puppet.pmtpa.wmflabs is DOWN address: 10.4.0.34 CRITICAL - Host Unreachable (10.4.0.34) [22:51:42] PROBLEM host: npm-packaging.pmtpa.wmflabs is DOWN address: 10.4.0.198 CRITICAL - Host Unreachable (10.4.0.198) [22:51:52] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [22:51:52] PROBLEM host: ve-roundtrip2.pmtpa.wmflabs is DOWN address: 10.4.0.162 CRITICAL - Host Unreachable (10.4.0.162) [22:52:02] PROBLEM host: bastion3.pmtpa.wmflabs is DOWN address: 10.4.1.85 CRITICAL - Host Unreachable (10.4.1.85) [22:52:12] PROBLEM host: dumpmuncher.pmtpa.wmflabs is DOWN address: 10.4.0.32 CRITICAL - Host Unreachable (10.4.0.32) [22:52:22] PROBLEM host: lucene-test1.pmtpa.wmflabs is DOWN address: 10.4.0.205 CRITICAL - Host Unreachable (10.4.0.205) [22:52:22] PROBLEM host: babbage0.pmtpa.wmflabs is DOWN address: 10.4.1.51 CRITICAL - Host Unreachable (10.4.1.51) [22:52:22] PROBLEM host: sf-book-server.pmtpa.wmflabs is DOWN address: 10.4.1.87 CRITICAL - Host Unreachable (10.4.1.87) [22:52:32] PROBLEM host: worker1.pmtpa.wmflabs is DOWN address: 10.4.1.12 CRITICAL - Host Unreachable (10.4.1.12) [22:52:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:52:52] PROBLEM host: dumps-2.pmtpa.wmflabs is DOWN address: 10.4.1.36 CRITICAL - Host Unreachable (10.4.1.36) [22:52:52] PROBLEM host: dumps-bot3.pmtpa.wmflabs is DOWN address: 10.4.0.118 CRITICAL - Host Unreachable (10.4.0.118) [22:52:52] PROBLEM host: nagios-dev.pmtpa.wmflabs is DOWN address: 10.4.0.201 CRITICAL - Host Unreachable (10.4.0.201) [22:53:02] PROBLEM host: lynwood.pmtpa.wmflabs is DOWN address: 10.4.0.2 CRITICAL - Host Unreachable (10.4.0.2) [22:53:02] PROBLEM host: node.pmtpa.wmflabs is DOWN address: 10.4.1.63 CRITICAL - Host Unreachable (10.4.1.63) [22:53:02] PROBLEM host: redmine.pmtpa.wmflabs is DOWN address: 10.4.1.71 CRITICAL - Host Unreachable (10.4.1.71) [22:53:12] PROBLEM host: wep2.pmtpa.wmflabs is DOWN address: 10.4.0.57 CRITICAL - Host Unreachable (10.4.0.57) [22:53:22] PROBLEM host: authorlinks.pmtpa.wmflabs is DOWN address: 10.4.1.62 CRITICAL - Host Unreachable (10.4.1.62) [22:53:22] PROBLEM host: bots-abogott-devel.pmtpa.wmflabs is DOWN address: 10.4.1.42 CRITICAL - Host Unreachable (10.4.1.42) [22:53:32] PROBLEM host: eventstream.pmtpa.wmflabs is DOWN address: 10.4.1.27 CRITICAL - Host Unreachable (10.4.1.27) [22:53:32] PROBLEM host: gerrit-build.pmtpa.wmflabs is DOWN address: 10.4.0.70 CRITICAL - Host Unreachable (10.4.0.70) [22:53:42] PROBLEM host: robh-mingle.pmtpa.wmflabs is DOWN address: 10.4.0.92 CRITICAL - Host Unreachable (10.4.0.92) [22:53:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:53:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:54:12] PROBLEM host: bots-analytics.pmtpa.wmflabs is DOWN address: 10.4.1.44 CRITICAL - Host Unreachable (10.4.1.44) [22:54:22] PROBLEM host: webtools-login.pmtpa.wmflabs is DOWN address: 10.4.0.24 CRITICAL - Host Unreachable (10.4.0.24) [22:54:22] PROBLEM host: webtools-odie.pmtpa.wmflabs is DOWN address: 10.4.1.39 CRITICAL - Host Unreachable (10.4.1.39) [22:54:32] PROBLEM host: checkwiki-web3.pmtpa.wmflabs is DOWN address: 10.4.1.74 CRITICAL - Host Unreachable (10.4.1.74) [22:54:32] PROBLEM host: damian-keytest.pmtpa.wmflabs is DOWN address: 10.4.0.97 CRITICAL - Host Unreachable (10.4.0.97) [22:54:55] PROBLEM host: bots-nr2.pmtpa.wmflabs is DOWN address: 10.4.1.66 CRITICAL - Host Unreachable (10.4.1.66) [22:55:42] PROBLEM host: deployment-puptest.pmtpa.wmflabs is DOWN address: 10.4.0.195 CRITICAL - Host Unreachable (10.4.0.195) [22:57:52] PROBLEM host: puppet1.pmtpa.wmflabs is DOWN address: 10.4.0.251 CRITICAL - Host Unreachable (10.4.0.251) [22:57:52] PROBLEM host: maps-osmmapnik.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [22:57:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 6% free memory [23:03:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:03:53] PROBLEM Current Load is now: CRITICAL on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: Connection refused by host [23:04:33] PROBLEM Disk Space is now: CRITICAL on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: Connection refused by host [23:05:12] PROBLEM Free ram is now: CRITICAL on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: Connection refused by host [23:05:42] PROBLEM host: build2.pmtpa.wmflabs is DOWN address: 10.4.0.161 CRITICAL - Host Unreachable (10.4.0.161) [23:06:42] PROBLEM Total processes is now: CRITICAL on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: Connection refused by host [23:07:22] PROBLEM dpkg-check is now: CRITICAL on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: Connection refused by host [23:09:22] PROBLEM host: deployment-integration.pmtpa.wmflabs is DOWN address: 10.4.1.61 CRITICAL - Host Unreachable (10.4.1.61) [23:09:32] RECOVERY Disk Space is now: OK on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: DISK OK [23:10:13] RECOVERY Free ram is now: OK on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: OK: 94% free memory [23:11:43] RECOVERY Total processes is now: OK on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: PROCS OK: 90 processes [23:12:23] RECOVERY dpkg-check is now: OK on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: All packages OK [23:13:53] RECOVERY Current Load is now: OK on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: OK - load average: 0.01, 0.41, 0.39 [23:16:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [23:19:12] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [23:19:22] PROBLEM host: maps-pgsql3.pmtpa.wmflabs is DOWN address: 10.4.1.75 CRITICAL - Host Unreachable (10.4.1.75) [23:19:32] PROBLEM host: incubator-mc.pmtpa.wmflabs is DOWN address: 10.4.0.20 CRITICAL - Host Unreachable (10.4.0.20) [23:19:42] PROBLEM host: gadgets-apache3.pmtpa.wmflabs is DOWN address: 10.4.1.92 CRITICAL - Host Unreachable (10.4.1.92) [23:19:42] PROBLEM host: mingle.pmtpa.wmflabs is DOWN address: 10.4.0.55 CRITICAL - Host Unreachable (10.4.0.55) [23:19:52] PROBLEM host: mwreview.pmtpa.wmflabs is DOWN address: 10.4.0.30 CRITICAL - Host Unreachable (10.4.0.30) [23:19:52] PROBLEM host: new-ns0.pmtpa.wmflabs is DOWN address: 10.4.0.190 CRITICAL - Host Unreachable (10.4.0.190) [23:19:52] PROBLEM host: mobile-pg.pmtpa.wmflabs is DOWN address: 10.4.1.46 CRITICAL - Host Unreachable (10.4.1.46) [23:19:52] PROBLEM host: generator.pmtpa.wmflabs is DOWN address: 10.4.0.189 CRITICAL - Host Unreachable (10.4.0.189) [23:20:23] PROBLEM host: fundraising-ci.pmtpa.wmflabs is DOWN address: 10.4.0.221 CRITICAL - Host Unreachable (10.4.0.221) [23:20:33] PROBLEM host: gerrit-dev.pmtpa.wmflabs is DOWN address: 10.4.0.207 CRITICAL - Host Unreachable (10.4.0.207) [23:20:33] PROBLEM host: integration-contintrefactor.pmtpa.wmflabs is DOWN address: 10.4.1.52 CRITICAL - Host Unreachable (10.4.1.52) [23:20:53] PROBLEM host: gerrit-db.pmtpa.wmflabs is DOWN address: 10.4.0.47 CRITICAL - Host Unreachable (10.4.0.47) [23:21:23] PROBLEM host: nginx-devunt.pmtpa.wmflabs is DOWN address: 10.4.0.65 CRITICAL - Host Unreachable (10.4.0.65) [23:21:43] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [23:21:43] PROBLEM host: glam-gwtoolset-puppet.pmtpa.wmflabs is DOWN address: 10.4.0.34 CRITICAL - Host Unreachable (10.4.0.34) [23:21:43] PROBLEM host: search1.pmtpa.wmflabs is DOWN address: 10.4.0.100 CRITICAL - Host Unreachable (10.4.0.100) [23:21:53] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [23:21:53] PROBLEM host: ve-roundtrip2.pmtpa.wmflabs is DOWN address: 10.4.0.162 CRITICAL - Host Unreachable (10.4.0.162) [23:22:03] PROBLEM host: bastion3.pmtpa.wmflabs is DOWN address: 10.4.1.85 CRITICAL - Host Unreachable (10.4.1.85) [23:22:03] PROBLEM host: gerrit-dev-zap.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [23:22:03]