[00:17:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:19:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:23:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:24:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:24:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:25:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:29:55] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:38:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:40:54] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [00:41:24] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [00:46:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [00:47:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:49:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:54:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:54:14] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:54:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:56:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:59:32] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 16% free memory [01:00:22] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:03:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [01:17:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:19:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:24:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:24:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:26:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:26:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:31:23] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:36:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:41:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [01:47:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:48:53] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 23% free memory [01:49:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:54:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:54:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:56:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:56:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:02:22] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:05:33] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK CRITICAL - free space: / 223 MB (2% inode=77%): [02:17:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:19:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:24:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:24:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:26:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:26:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:33:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:37:53] RECOVERY Total processes is now: OK on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS OK: 149 processes [02:40:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [02:47:39] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:49:49] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:54:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:54:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:55:38] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 189 processes [02:56:38] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:57:38] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:00:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [03:03:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:06:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [03:17:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:20:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:21:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [03:24:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:24:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:27:17] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:28:27] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:33:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:47:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:50:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:54:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:54:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:57:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:58:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:03:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:05:13] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [04:17:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:20:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:21:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [04:24:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:24:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:28:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:29:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:33:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:35:13] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 20% free memory [04:38:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [04:40:54] PROBLEM Free ram is now: WARNING on sultest2.pmtpa.wmflabs 10.4.0.200 output: Warning: 17% free memory [04:41:44] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:48:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:50:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:54:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:54:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:56:22] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [04:56:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [04:59:02] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:59:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:03:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:05:53] RECOVERY Free ram is now: OK on sultest2.pmtpa.wmflabs 10.4.0.200 output: OK: 25% free memory [05:18:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:21:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:24:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:26:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:26:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [05:29:03] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:29:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:33:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:48:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:51:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:51:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [05:56:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:57:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:59:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:59:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:04:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:06:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [06:19:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:21:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:26:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:27:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:28:13] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [06:29:14] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:29:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:29:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:30:52] PROBLEM Total processes is now: WARNING on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS WARNING: 151 processes [06:34:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:35:53] RECOVERY Total processes is now: OK on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS OK: 149 processes [06:39:13] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 146 processes [06:43:13] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 20% free memory [06:46:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 8% free memory [06:49:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:51:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:57:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:58:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:00:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:01:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:04:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:19:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:24:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:26:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [07:27:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:28:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:30:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 186 processes [07:30:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:31:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:34:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:49:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:54:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:57:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:58:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:00:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:01:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:04:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:20:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:24:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:27:08] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:28:58] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:31:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:34:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:39:32] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [08:41:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [08:47:23] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [08:51:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:54:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:59:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:59:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:00:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:01:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:04:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:04:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:22:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:24:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:29:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:29:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:31:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:34:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:52:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:54:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:59:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:59:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:01:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:02:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:07:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:22:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:24:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:29:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:31:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:31:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:34:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:38:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:52:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:54:22] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:58:42] PROBLEM Disk Space is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: DISK WARNING - free space: / 525 MB (5% inode=80%): [10:59:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:01:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:02:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:04:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:06:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [11:08:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:11:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [11:22:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:24:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:29:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:31:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:32:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:34:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:38:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:43:43] PROBLEM Disk Space is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: DISK CRITICAL - free space: / 239 MB (2% inode=80%): [11:51:46] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:52:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:54:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:58:52] PROBLEM dpkg-check is now: CRITICAL on wiktionary-dev.pmtpa.wmflabs 10.4.0.215 output: DPKG CRITICAL dpkg reports broken packages [12:00:43] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:02:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:03:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:03:53] RECOVERY dpkg-check is now: OK on wiktionary-dev.pmtpa.wmflabs 10.4.0.215 output: All packages OK [12:04:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:08:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:23:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:25:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:26:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [12:31:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:32:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:33:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:34:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:38:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:39:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:41:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [12:47:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [12:50:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 203 processes [12:53:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:54:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [12:55:36] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 192 processes [12:55:46] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:02:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:03:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:03:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:04:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:08:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:23:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:26:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:26:42] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [13:31:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:32:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:33:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:34:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:38:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:53:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:56:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:00:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [14:02:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:03:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:03:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:04:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:08:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:09:22] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 151 processes [14:19:22] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 147 processes [14:23:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:25:37] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 181 processes [14:26:37] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:33:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:34:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:34:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:35:13] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:35:53] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:36:23] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:36:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:38:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:40:54] RECOVERY Total processes is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS OK: 135 processes [14:40:54] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [14:41:24] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.10, 0.70, 0.87 [14:41:24] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 25% free memory [14:54:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:58:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:01:42] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [15:03:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:04:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:04:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:06:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:07:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:08:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:24:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:27:02] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [15:29:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:33:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:34:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:37:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:38:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:51:42] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [15:54:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:59:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:04:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:04:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:08:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:09:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:24:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:26:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [16:29:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:32:03] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 22% free memory [16:34:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:34:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:36:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:37:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:39:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:39:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:39:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:47:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [16:54:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:59:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:00:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:04:22] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:04:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:08:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:09:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:09:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:24:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:26:43] PROBLEM Current Load is now: WARNING on bots-sql1.pmtpa.wmflabs 10.4.0.52 output: WARNING - load average: 9.45, 9.70, 6.76 [17:29:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:31:42] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [17:34:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:34:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:36:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [17:38:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:39:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:39:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:46:43] RECOVERY Current Load is now: OK on bots-sql1.pmtpa.wmflabs 10.4.0.52 output: OK - load average: 1.99, 3.42, 4.94 [17:54:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:59:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:04:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:04:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:08:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:09:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:09:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:21:43] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [18:24:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:29:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:34:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:36:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:38:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:39:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:40:05] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:54:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:56:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [18:59:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:06:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:07:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:08:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:09:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:11:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:24:34] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:29:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:36:59] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:38:06] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:38:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:39:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:41:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:56:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:00:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:06:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:06:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [20:07:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:08:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:09:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:11:35] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:11:45] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [20:26:58] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:30:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:37:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [20:38:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:39:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:40:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:40:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [20:42:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:42:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:45:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 182 processes [20:45:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 19% free memory [20:48:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [20:57:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [21:00:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [21:01:46] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:02:32] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [21:07:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [21:08:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [21:09:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:10:25] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 15% free memory [21:13:05] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [21:14:25] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [21:27:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [21:30:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [21:31:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [21:37:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [21:38:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [21:38:53] PROBLEM Current Load is now: CRITICAL on frack-proxy.pmtpa.wmflabs 10.4.0.226 output: Connection refused by host [21:39:33] PROBLEM Disk Space is now: CRITICAL on frack-proxy.pmtpa.wmflabs 10.4.0.226 output: Connection refused by host [21:39:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:40:12] PROBLEM Free ram is now: CRITICAL on frack-proxy.pmtpa.wmflabs 10.4.0.226 output: Connection refused by host [21:41:42] PROBLEM Total processes is now: CRITICAL on frack-proxy.pmtpa.wmflabs 10.4.0.226 output: Connection refused by host [21:42:22] PROBLEM dpkg-check is now: CRITICAL on frack-proxy.pmtpa.wmflabs 10.4.0.226 output: Connection refused by host [21:43:42]