[00:04:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:18:13] RECOVERY host: generator.pmtpa.wmflabs is UP address: 10.4.0.189 PING OK - Packet loss = 0%, RTA = 0.76 ms [00:18:23] RECOVERY host: babbage0.pmtpa.wmflabs is UP address: 10.4.1.51 PING OK - Packet loss = 0%, RTA = 0.74 ms [00:18:43] RECOVERY host: lynwood.pmtpa.wmflabs is UP address: 10.4.0.2 PING OK - Packet loss = 0%, RTA = 0.70 ms [00:18:43] RECOVERY host: parsoid-roundtrip7-8core.pmtpa.wmflabs is UP address: 10.4.1.26 PING OK - Packet loss = 0%, RTA = 0.67 ms [00:19:13] PROBLEM host: saltmaster.pmtpa.wmflabs is DOWN address: 10.4.0.150 CRITICAL - Host Unreachable (10.4.0.150) [00:19:13] RECOVERY host: search1.pmtpa.wmflabs is UP address: 10.4.0.100 PING OK - Packet loss = 0%, RTA = 0.57 ms [00:19:53] RECOVERY host: mwreview.pmtpa.wmflabs is UP address: 10.4.0.30 PING OK - Packet loss = 0%, RTA = 0.62 ms [00:19:53] RECOVERY host: nginx-devunt.pmtpa.wmflabs is UP address: 10.4.0.65 PING OK - Packet loss = 0%, RTA = 0.74 ms [00:21:02] RECOVERY host: puppet1.pmtpa.wmflabs is UP address: 10.4.0.251 PING OK - Packet loss = 0%, RTA = 3.22 ms [00:22:12] PROBLEM host: testing-bootstrap5.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [00:22:12] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 0.99, 0.72, 0.34 [00:22:37] PROBLEM host: sf-book-server.pmtpa.wmflabs is DOWN address: 10.4.1.87 CRITICAL - Host Unreachable (10.4.1.87) [00:22:52] RECOVERY host: robh-mingle.pmtpa.wmflabs is UP address: 10.4.0.92 PING OK - Packet loss = 0%, RTA = 0.73 ms [00:22:52] RECOVERY host: redmine.pmtpa.wmflabs is UP address: 10.4.1.71 PING OK - Packet loss = 0%, RTA = 0.78 ms [00:23:02] PROBLEM host: worker1.pmtpa.wmflabs is DOWN address: 10.4.1.12 CRITICAL - Host Unreachable (10.4.1.12) [00:23:12] RECOVERY host: sf-book-server.pmtpa.wmflabs is UP address: 10.4.1.87 PING OK - Packet loss = 0%, RTA = 0.64 ms [00:23:52] PROBLEM host: gerrit-build.pmtpa.wmflabs is DOWN address: 10.4.0.70 CRITICAL - Host Unreachable (10.4.0.70) [00:24:02] RECOVERY host: stackfarm-sql2.pmtpa.wmflabs is UP address: 10.4.1.23 PING OK - Packet loss = 0%, RTA = 0.64 ms [00:24:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:24:22] RECOVERY host: saltmaster.pmtpa.wmflabs is UP address: 10.4.0.150 PING OK - Packet loss = 0%, RTA = 0.73 ms [00:24:42] PROBLEM host: wep2.pmtpa.wmflabs is DOWN address: 10.4.0.57 CRITICAL - Host Unreachable (10.4.0.57) [00:26:03] RECOVERY host: wep2.pmtpa.wmflabs is UP address: 10.4.0.57 PING OK - Packet loss = 0%, RTA = 3.57 ms [00:26:23] RECOVERY host: tstarling-puppet.pmtpa.wmflabs is UP address: 10.4.1.79 PING OK - Packet loss = 0%, RTA = 4.08 ms [00:26:43] RECOVERY host: worker1.pmtpa.wmflabs is UP address: 10.4.1.12 PING OK - Packet loss = 0%, RTA = 6.11 ms [00:26:53] RECOVERY host: gerrit-build.pmtpa.wmflabs is UP address: 10.4.0.70 PING OK - Packet loss = 0%, RTA = 3.29 ms [00:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [00:37:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 6% free memory [00:40:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [00:40:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [00:41:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:47:52] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 4% free memory [00:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:05:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [01:09:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [01:13:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [01:20:23] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [01:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:33:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:47:52] RECOVERY Disk Space is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: DISK OK [01:47:52] RECOVERY dpkg-check is now: OK on patchtest.pmtpa.wmflabs 10.4.0.69 output: All packages OK [01:48:02] RECOVERY Current Load is now: OK on patchtest.pmtpa.wmflabs 10.4.0.69 output: OK - load average: 0.00, 0.00, 0.00 [01:48:12] RECOVERY Total processes is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: PROCS OK: 87 processes [01:48:32] RECOVERY Disk Space is now: OK on patchtest.pmtpa.wmflabs 10.4.0.69 output: DISK OK [01:48:32] RECOVERY Free ram is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: OK: 75% free memory [01:49:12] RECOVERY Free ram is now: OK on patchtest.pmtpa.wmflabs 10.4.0.69 output: OK: 77% free memory [01:49:12] RECOVERY dpkg-check is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: All packages OK [01:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:56:02] PROBLEM Current Load is now: CRITICAL on patchtest.pmtpa.wmflabs 10.4.0.69 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:56:02] PROBLEM dpkg-check is now: CRITICAL on patchtest.pmtpa.wmflabs 10.4.0.69 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:56:02] PROBLEM Disk Space is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:56:22] PROBLEM Total processes is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:56:42] PROBLEM Disk Space is now: CRITICAL on patchtest.pmtpa.wmflabs 10.4.0.69 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:56:42] PROBLEM Free ram is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:57:22] PROBLEM Free ram is now: CRITICAL on patchtest.pmtpa.wmflabs 10.4.0.69 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:57:22] PROBLEM dpkg-check is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:57:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 7% free memory [01:58:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [02:04:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:17:52] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 4% free memory [02:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:26:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [02:34:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:04:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:20:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [03:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:34:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:53:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [03:55:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:04:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:11:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 11.24, 9.67, 6.67 [04:11:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 11.66, 9.72, 6.35 [04:13:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 12.87, 11.25, 7.51 [04:15:13] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 10.46, 9.68, 6.87 [04:16:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.66, 9.19, 6.63 [04:17:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.45, 10.01, 7.25 [04:26:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [04:26:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:27:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 7% free memory [04:34:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [04:34:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:38:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [04:39:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [04:39:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [04:40:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [04:52:55] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [04:56:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:57:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [04:58:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [05:04:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:11:23] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [05:28:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:34:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:42:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [05:51:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [05:58:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:04:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:06:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 4.02, 3.15, 4.98 [06:08:52] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 1.82, 1.90, 4.66 [06:12:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 3.46, 3.54, 4.75 [06:28:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:29:24] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [06:34:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:40:13] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 0.32, 2.13, 4.86 [06:46:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 0.37, 0.97, 3.73 [06:49:53] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 16% free memory [06:56:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 0.41, 1.78, 3.80 [06:57:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [06:57:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 7% free memory [06:59:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:04:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:15:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [07:22:53] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [07:29:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:34:45] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:44:52] PROBLEM Free ram is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Critical: 4% free memory [07:49:53] RECOVERY Free ram is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK: 95% free memory [07:59:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:04:54] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:19:24] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [08:30:00] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:34:54] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:37:17] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [08:37:57] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 7% free memory [08:38:47] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [08:40:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [08:42:52] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [08:47:24] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [08:56:46] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [08:58:16] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [08:59:56] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:00:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:06:55] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:27:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 8% free memory [09:28:14] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [09:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:36:55] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:37:35] RECOVERY Disk Space is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: DISK OK [09:42:22] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [09:47:25] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [10:00:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:02:54] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [10:06:57] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:16:14] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [10:30:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:39:26] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:47:15] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [11:00:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:09:35] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:12:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 6% free memory [11:20:23] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [11:31:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:39:35] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:55:45] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 146 processes [11:56:15] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [12:01:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:03:53] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 21% free memory [12:09:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:18:46] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 166 processes [12:20:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 155 processes [12:25:35] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [12:29:15] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [12:33:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:39:14] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [12:39:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:40:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:41:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [12:46:55] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [13:03:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [13:03:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:03:42] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 143 processes [13:09:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:09:54] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [13:11:54] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 16% free memory [13:20:24] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [13:32:15] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [13:33:25] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:39:44] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:48:16] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [14:00:42] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 154 processes [14:01:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 158 processes [14:03:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:06:37] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 140 processes [14:09:17] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [14:09:47] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:10:44] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 143 processes [14:15:44] PROBLEM SSH is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.0.54 output: Server answer: [14:27:14] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [14:33:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:39:25] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [14:40:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [14:41:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:53:46] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 161 processes [15:03:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:09:44] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [15:14:22] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:34:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:41:44] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [15:44:24] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:48:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [16:04:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:12:03] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [16:14:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:21:16] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [16:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:38:16] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:39:46] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:40:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [16:41:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [16:42:03] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [16:44:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:47:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [16:51:24] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:05:46] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:07:26] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [17:08:16] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [17:09:26] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [17:12:52] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [17:14:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:36:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:42:53] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [17:44:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:45:35] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 167 processes [18:02:23] PROBLEM Free ram is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:06:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:07:16] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [18:14:12] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [18:14:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:21:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [18:36:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:44:12] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [18:44:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:47:25] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [18:49:15] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [19:03:55] PROBLEM Current Load is now: CRITICAL on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [19:04:35] PROBLEM Disk Space is now: CRITICAL on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [19:05:14] PROBLEM Free ram is now: CRITICAL on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [19:06:44] PROBLEM Total processes is now: CRITICAL on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [19:06:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:07:24] PROBLEM dpkg-check is now: CRITICAL on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: Connection refused by host [19:09:34] RECOVERY Disk Space is now: OK on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: DISK OK [19:10:14] RECOVERY Free ram is now: OK on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: OK: 73% free memory [19:11:44] RECOVERY Total processes is now: OK on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: PROCS OK: 84 processes [19:12:24] RECOVERY dpkg-check is now: OK on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: All packages OK [19:13:54] RECOVERY Current Load is now: OK on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: OK - load average: 0.03, 0.47, 0.48 [19:14:14] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [19:14:54] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:20:43] RECOVERY SSH is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [19:25:23] PROBLEM dpkg-check is now: CRITICAL on maps-test030613-1.pmtpa.wmflabs 10.4.0.45 output: DPKG CRITICAL dpkg reports broken packages [19:28:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [19:37:33] PROBLEM Free ram is now: WARNING on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: Warning: 19% free memory [19:37:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:41:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [19:42:32] RECOVERY Free ram is now: OK on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: OK: 36% free memory [19:44:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [19:46:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:57:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [19:58:54] PROBLEM Current Load is now: CRITICAL on tools-db.pmtpa.wmflabs 10.4.0.8 output: Connection refused by host [19:59:34] PROBLEM Disk Space is now: CRITICAL on tools-db.pmtpa.wmflabs 10.4.0.8 output: Connection refused by host [20:00:22] PROBLEM Free ram is now: CRITICAL on tools-db.pmtpa.wmflabs 10.4.0.8 output: Connection refused by host [20:01:42] PROBLEM Total processes is now: CRITICAL on tools-db.pmtpa.wmflabs 10.4.0.8 output: Connection refused by host [20:02:22] PROBLEM dpkg-check is now: CRITICAL on tools-db.pmtpa.wmflabs 10.4.0.8 output: Connection refused by host [20:03:52] RECOVERY Current Load is now: OK on tools-db.pmtpa.wmflabs 10.4.0.8 output: OK - load average: 0.50, 0.85, 0.52 [20:04:32] RECOVERY Disk Space is now: OK on tools-db.pmtpa.wmflabs 10.4.0.8 output: DISK OK [20:05:19] RECOVERY Free ram is now: OK on tools-db.pmtpa.wmflabs 10.4.0.8 output: OK: 97% free memory [20:06:43] RECOVERY Total processes is now: OK on tools-db.pmtpa.wmflabs 10.4.0.8 output: PROCS OK: 119 processes [20:07:23] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [20:07:23] RECOVERY dpkg-check is now: OK on tools-db.pmtpa.wmflabs 10.4.0.8 output: All packages OK [20:08:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:10:48] PROBLEM Free ram is now: CRITICAL on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: Critical: 2% free memory [20:14:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [20:17:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:28:52] PROBLEM Current Load is now: CRITICAL on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: Connection refused by host [20:29:32] PROBLEM Disk Space is now: CRITICAL on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: Connection refused by host [20:30:13] PROBLEM Free ram is now: CRITICAL on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: Connection refused by host [20:31:43] PROBLEM Total processes is now: CRITICAL on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: Connection refused by host [20:32:23] PROBLEM dpkg-check is now: CRITICAL on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: Connection refused by host [20:33:53] RECOVERY Current Load is now: OK on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: OK - load average: 0.92, 0.98, 0.57 [20:34:33] RECOVERY Disk Space is now: OK on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: DISK OK [20:35:12] RECOVERY Free ram is now: OK on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: OK: 97% free memory [20:36:42] RECOVERY Total processes is now: OK on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: PROCS OK: 120 processes [20:37:22] RECOVERY dpkg-check is now: OK on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: All packages OK [20:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [20:38:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:39:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [20:41:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [20:41:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:44:23] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [20:45:22] PROBLEM dpkg-check is now: CRITICAL on deployment-searchidx02.pmtpa.wmflabs 10.4.0.84 output: DPKG CRITICAL dpkg reports broken packages [20:47:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:55:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [21:04:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [21:05:32] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 148 processes [21:08:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:13:35] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 160 processes [21:14:15] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [21:14:25] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [21:19:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:20:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [21:38:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:44:42] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [21:49:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:59:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [22:08:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:14:42] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [22:19:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:27:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [22:39:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:43:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [22:44:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [22:49:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:09:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:14:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [23:19:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:20:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [23:39:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:44:45] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [23:48:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [23:49:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [23:58:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 194 processes