[00:19:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 207 processes [00:22:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:24:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:24:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:24:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:24:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:24:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:36:23] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [00:39:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 183 processes [00:40:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 21% free memory [00:40:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [00:48:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [00:48:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 19% free memory [00:52:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:54:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:54:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:54:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:54:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:54:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:55:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:00:54] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [01:04:34] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [01:22:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:24:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:24:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:24:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:24:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:24:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:25:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:40:43] @statistics-on [01:40:44] Statistics were now enabled [01:44:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 186 processes [01:53:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:54:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:54:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:55:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:56:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:57:23] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:58:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:23:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:24:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:24:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:26:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:27:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:27:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:28:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:53:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:54:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:54:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:57:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:57:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:58:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:58:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:59:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 186 processes [03:17:15] PROBLEM Disk Space is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: Connection refused by host [03:17:25] PROBLEM Current Load is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: Connection refused by host [03:17:55] PROBLEM Disk Space is now: CRITICAL on tools-master.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [03:17:55] PROBLEM Total processes is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: Connection refused by host [03:17:55] PROBLEM dpkg-check is now: CRITICAL on tools-master.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [03:17:55] PROBLEM Free ram is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: Connection refused by host [03:18:05] PROBLEM Current Load is now: CRITICAL on tools-master.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [03:18:15] PROBLEM Total processes is now: CRITICAL on tools-master.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [03:18:35] PROBLEM Free ram is now: CRITICAL on tools-master.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [03:18:35] PROBLEM dpkg-check is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: Connection refused by host [03:23:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:23:33] RECOVERY dpkg-check is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: All packages OK [03:24:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:25:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:27:12] RECOVERY Disk Space is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: DISK OK [03:27:32] RECOVERY Current Load is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: OK - load average: 0.02, 0.46, 0.45 [03:27:52] RECOVERY Disk Space is now: OK on tools-master.pmtpa.wmflabs 10.4.1.54 output: DISK OK [03:27:52] RECOVERY Total processes is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: PROCS OK: 83 processes [03:27:52] RECOVERY Free ram is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.0.128 output: OK: 91% free memory [03:27:52] RECOVERY dpkg-check is now: OK on tools-master.pmtpa.wmflabs 10.4.1.54 output: All packages OK [03:28:12] RECOVERY Current Load is now: OK on tools-master.pmtpa.wmflabs 10.4.1.54 output: OK - load average: 0.13, 0.73, 0.56 [03:28:12] RECOVERY Total processes is now: OK on tools-master.pmtpa.wmflabs 10.4.1.54 output: PROCS OK: 83 processes [03:28:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:28:32] RECOVERY Free ram is now: OK on tools-master.pmtpa.wmflabs 10.4.1.54 output: OK: 91% free memory [03:28:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:29:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:29:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:54:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:55:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:56:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:58:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:58:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:59:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:59:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:24:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:25:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:26:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:28:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:28:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:29:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:29:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:38:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [04:48:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 9.97, 8.46, 6.38 [04:52:03] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 3.68, 6.80, 5.44 [04:54:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:55:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:56:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:57:02] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.53, 4.80, 4.95 [04:58:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:58:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:59:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:59:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:05:12] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.30, 7.62, 6.13 [05:06:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [05:06:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.66, 8.21, 6.30 [05:10:32] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 14% free memory [05:24:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:25:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:26:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:28:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:28:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:29:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:29:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:55:45] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:55:53] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:56:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:58:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:58:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:59:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:59:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:25:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:25:53] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:26:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:28:13] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:28:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:28:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:29:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:29:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:30:33] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 30% free memory [06:33:13] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 147 processes [06:48:42] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 1.12, 1.61, 4.70 [06:56:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:56:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:57:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:58:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:59:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:59:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:59:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:06:01] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.24, 1.64, 4.06 [07:06:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 0.63, 0.82, 3.89 [07:26:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:26:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:27:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:28:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:29:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:29:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:29:54] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:56:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:57:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:57:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:58:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:59:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:59:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:59:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:27:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:28:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:29:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:29:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:29:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:30:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:32:25] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:37:34] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 22% free memory [08:38:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [08:57:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:59:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:59:16] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:59:16] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:59:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:00:35] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 16% free memory [09:00:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:01:46] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [09:03:06] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:15:23] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes [09:27:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:29:07] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:29:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:29:16] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:29:44] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:30:22] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 148 processes [09:31:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:33:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:57:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:59:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:59:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:59:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:01:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:01:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:03:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:08:22] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [10:27:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:29:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:29:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:29:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:31:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:31:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:33:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:57:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:59:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:59:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:59:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:01:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:01:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:03:23] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 153 processes [11:03:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:08:22] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 149 processes [11:27:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:29:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:29:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:29:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:31:27] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:31:37] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:33:47] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:57:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:59:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:59:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:59:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:02:15] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:03:25] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:03:55] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:27:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:29:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:29:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:29:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:33:02] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:34:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:40:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [12:41:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:41:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [12:48:28] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 18% free memory [12:57:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:59:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:59:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:59:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [12:59:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:03:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:03:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:04:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:04:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [13:27:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:29:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:29:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:29:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:34:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:34:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:38:53] RECOVERY dpkg-check is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: All packages OK [13:58:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:59:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:59:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:02:45] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:03:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:04:25] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:04:25] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:28:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:29:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:31:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:32:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:34:02] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:34:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:34:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:58:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:59:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:01:36] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:03:26] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:04:06] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:04:36] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:04:56] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:28:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:29:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:32:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:33:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:34:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:34:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:37:58] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:58:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:59:44] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:02:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:03:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:04:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:04:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:08:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:29:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:30:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:32:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:32:53] PROBLEM Total processes is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: PROCS WARNING: 151 processes [16:33:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:34:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:37:52] RECOVERY Total processes is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: PROCS OK: 145 processes [16:38:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:39:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:39:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:52:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [16:52:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [16:59:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:01:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:03:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:04:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:06:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:08:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:31:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:31:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:33:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:34:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:35:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:36:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:38:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:44:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [17:59:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 192 processes [18:01:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:01:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:03:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:04:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:06:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:08:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:31:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:31:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:33:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:34:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:35:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:36:35] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:38:05] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:01:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:01:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:03:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:04:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:06:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:07:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:08:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:12:42] PROBLEM Free ram is now: CRITICAL on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Critical: 5% free memory [19:17:43] PROBLEM Free ram is now: WARNING on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Warning: 7% free memory [19:20:22] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 152 processes [19:31:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:32:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:33:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:34:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:36:27] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:37:17] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:38:47] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:54:23] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes [19:57:12] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:57:32] PROBLEM SSH is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Server answer: [19:57:52] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:58:22] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 23% free memory [19:59:22] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 150 processes [20:01:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:02:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:02:13] RECOVERY Total processes is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS OK: 136 processes [20:02:33] RECOVERY SSH is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [20:02:53] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [20:03:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:04:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:08:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:09:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:09:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:29:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [20:31:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:32:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:33:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:34:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:37:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [20:38:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:39:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:39:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:40:22] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [20:43:52] PROBLEM Current Load is now: CRITICAL on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [20:44:32] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [20:45:12] PROBLEM Free ram is now: CRITICAL on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [20:46:42] PROBLEM Total processes is now: CRITICAL on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [20:47:22] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: DPKG CRITICAL dpkg reports broken packages [20:48:52] RECOVERY Current Load is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: OK - load average: 0.36, 0.96, 0.63 [20:49:32] RECOVERY Disk Space is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: DISK OK [20:50:12] RECOVERY Free ram is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: OK: 89% free memory [20:50:27] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [20:51:42] RECOVERY Total processes is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: PROCS OK: 84 processes [20:52:22] RECOVERY dpkg-check is now: OK on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: All packages OK [21:00:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [21:02:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [21:04:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [21:04:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [21:04:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [21:08:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:09:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [21:09:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [21:15:22] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev3.pmtpa.wmflabs 10.4.0.246 output: DPKG CRITICAL dpkg reports broken packages [21:27:02] RECOVERY Current Load is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: OK - load average: 0.00, 0.00, 0.00 [21:27:12] RECOVERY Total processes is now: OK on patchtest.pmtpa.wmflabs 10.4.0.69 output: PROCS OK: 86 processes [21:32:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [21:34:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [21:34:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [21:34:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [21:34:33]