[00:16:18] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.73, 9.43, 6.51 [00:16:18] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 11.00, 9.34, 6.20 [00:16:19] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:16:19] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 10.12, 9.35, 6.71 [00:16:19] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:16:19] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:16:20] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:16:20] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:16:21] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:16:22] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:16:36] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:16:37] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:16:38] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:16:38] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:16:38] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:16:38] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:16:40] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:22:22] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 157 processes [00:33:06] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:34:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:38:36] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:39:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:39:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:40:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:40:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 27% free memory [00:41:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:43:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:58:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [01:04:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:04:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.45, 1.96, 4.93 [01:05:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:08:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:08:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 18% free memory [01:09:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:09:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:11:22] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [01:11:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:13:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:13:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.53, 3.40, 4.62 [01:20:52] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [01:34:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:35:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:38:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:39:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:39:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:41:36] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:43:46] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:46:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 0.61, 1.35, 4.46 [01:46:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 5.65, 5.73, 5.29 [01:56:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.32, 2.58, 4.05 [02:04:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:05:52] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:08:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:09:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:10:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:12:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:14:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:14:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 216 processes [02:34:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:36:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:38:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:39:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:40:47] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:42:17] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:43:27] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [02:44:27] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:48:22] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [03:04:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:06:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:08:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:09:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:11:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:14:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:14:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:14:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 194 processes [03:34:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:38:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:39:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:39:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:41:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:44:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:44:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:49:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [04:04:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:07:22] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 150 processes [04:09:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:09:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:09:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:11:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:14:26] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:14:56] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:24:37] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 194 processes [04:35:53] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:38:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [04:38:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [04:39:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:39:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:41:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:41:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:44:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:46:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [04:48:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:56:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [05:06:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:09:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:09:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:11:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:12:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:14:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:18:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:36:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:39:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:39:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:42:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:43:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:44:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:48:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:06:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:09:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:09:47] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:12:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:13:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:14:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:18:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:30:22] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 155 processes [06:31:12] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:31:52] PROBLEM Total processes is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: PROCS WARNING: 152 processes [06:35:23] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 147 processes [06:36:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:39:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:39:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:41:53] RECOVERY Total processes is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: PROCS OK: 147 processes [06:42:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:43:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:44:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:46:12] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 147 processes [06:48:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:07:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:09:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:11:06] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:12:06] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:13:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:16:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:18:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:31:43] PROBLEM Free ram is now: WARNING on bots-sql3.pmtpa.wmflabs 10.4.0.40 output: Warning: 18% free memory [07:36:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory [07:37:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:39:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:41:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:42:53] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:43:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:46:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:48:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:51:43] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [07:56:42] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 20% free memory [08:07:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:09:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:09:43] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 19% free memory [08:12:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:13:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:14:08] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:16:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:18:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:37:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:39:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:41:16] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [08:41:46] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 27% free memory [08:43:36] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:44:06] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:44:06] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:47:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:49:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:59:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [09:04:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:07:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:10:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:13:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:14:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:14:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:17:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:19:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:22:33] PROBLEM Total processes is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:23:33] PROBLEM Current Load is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:25:53] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 23% free memory [09:27:33] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 135 processes [09:28:33] RECOVERY Current Load is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK - load average: 0.16, 0.60, 0.74 [09:37:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:41:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:42:42] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 17% free memory [09:44:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:44:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:45:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:47:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:49:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:07:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:11:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:14:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:14:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:15:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:17:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:19:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:37:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:44:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:44:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:44:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:45:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:47:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:49:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:07:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:14:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:14:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:16:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:16:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:19:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:19:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:37:48] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:44:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:46:35] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:47:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:47:53] PROBLEM Total processes is now: WARNING on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS WARNING: 151 processes [11:48:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:49:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:49:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:08:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:14:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:17:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:18:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:18:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:19:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:22:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:38:07] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:38:27] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 22% free memory [12:39:18] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:44:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:46:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [12:47:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:48:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:48:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:49:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:52:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [12:53:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:09:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:14:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:18:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:18:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:19:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:19:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:23:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:29:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 203 processes [13:39:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:43:53] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [13:44:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:44:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [13:48:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:49:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:49:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:49:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:53:04] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:09:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:14:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:18:22] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 151 processes [14:18:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:19:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:19:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:19:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:23:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:28:22] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 150 processes [14:39:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:44:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:48:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:49:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:49:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:49:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:54:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:02:42] PROBLEM Free ram is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Critical: 5% free memory [15:06:22] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 154 processes [15:07:42] PROBLEM Free ram is now: WARNING on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Warning: 6% free memory [15:09:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:14:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:17:42] PROBLEM Free ram is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Critical: 5% free memory [15:19:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:19:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:20:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:21:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:24:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:39:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:44:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:49:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:49:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:51:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:52:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:54:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:10:53] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:14:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:19:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:19:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:22:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:23:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:24:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:37:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:39:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:41:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:45:44] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:49:44] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:50:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:52:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:53:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:54:54] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:55:17] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:07:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 18% free memory [17:11:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:11:23] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 142 processes [17:14:33] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [17:16:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:21:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:22:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:22:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:23:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:24:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:27:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 155 processes [17:30:33] PROBLEM dpkg-check is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: DPKG CRITICAL dpkg reports broken packages [17:35:32] RECOVERY dpkg-check is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: All packages OK [17:42:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:46:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:51:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:52:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:53:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:53:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:58:04] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:58:54] PROBLEM Current Load is now: CRITICAL on reportcard.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [17:59:34] PROBLEM Disk Space is now: CRITICAL on reportcard.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [18:03:53] RECOVERY Current Load is now: OK on reportcard.pmtpa.wmflabs 10.4.1.55 output: OK - load average: 0.04, 0.47, 0.41 [18:04:33] RECOVERY Disk Space is now: OK on reportcard.pmtpa.wmflabs 10.4.1.55 output: DISK OK [18:12:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:12:32] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 148 processes [18:16:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:21:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:22:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:23:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:24:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:28:08] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:42:25] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:43:55] PROBLEM Current Load is now: CRITICAL on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [18:43:55] PROBLEM Disk Space is now: CRITICAL on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: Connection refused by host [18:44:35] PROBLEM Disk Space is now: CRITICAL on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [18:44:35] PROBLEM Free ram is now: CRITICAL on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: Connection refused by host [18:45:23] PROBLEM Free ram is now: CRITICAL on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: Connection refused by host [18:45:53] PROBLEM Total processes is now: CRITICAL on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: Connection refused by host [18:46:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:48:53] RECOVERY Current Load is now: OK on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: OK - load average: 0.07, 0.59, 0.48 [18:48:53] RECOVERY Disk Space is now: OK on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: DISK OK [18:49:33] RECOVERY Disk Space is now: OK on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: DISK OK [18:49:33] RECOVERY Free ram is now: OK on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: OK: 94% free memory [18:50:22] RECOVERY Free ram is now: OK on deployment-search01.pmtpa.wmflabs 10.4.1.81 output: OK: 94% free memory [18:50:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 158 processes [18:50:52] RECOVERY Total processes is now: OK on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: PROCS OK: 90 processes [18:51:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:52:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:53:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:54:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:58:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:12:26] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:13:56] PROBLEM Current Load is now: CRITICAL on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [19:14:36] PROBLEM Disk Space is now: CRITICAL on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [19:15:23] PROBLEM Free ram is now: CRITICAL on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [19:16:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:16:43] PROBLEM Total processes is now: CRITICAL on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [19:17:33] PROBLEM dpkg-check is now: CRITICAL on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: Connection refused by host [19:18:53] RECOVERY Current Load is now: OK on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: OK - load average: 1.30, 1.16, 0.63 [19:19:33] RECOVERY Disk Space is now: OK on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: DISK OK [19:20:22] RECOVERY Free ram is now: OK on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: OK: 80% free memory [19:21:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:21:42] RECOVERY Total processes is now: OK on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: PROCS OK: 93 processes [19:22:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:23:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:24:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:28:48] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:39:43] PROBLEM dpkg-check is now: CRITICAL on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: DPKG CRITICAL dpkg reports broken packages [19:42:32] RECOVERY dpkg-check is now: OK on maps-test010313.pmtpa.wmflabs 10.4.1.84 output: All packages OK [19:42:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:46:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:51:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:52:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:53:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:54:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:54:42] RECOVERY dpkg-check is now: OK on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: All packages OK [19:59:06] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:12:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:18:22] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:21:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:23:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:23:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:24:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:29:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [20:40:15] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:42:45] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:45:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [20:47:42] PROBLEM dpkg-check is now: CRITICAL on deployment-searchidx01.pmtpa.wmflabs 10.4.0.56 output: DPKG CRITICAL dpkg reports broken packages [20:49:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:50:23]