[00:12:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [00:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [00:39:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [00:41:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [00:42:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [00:59:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [01:07:53] PROBLEM Free ram is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: Critical: 5% free memory [01:10:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [01:12:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [01:12:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [01:37:52] RECOVERY Free ram is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: OK: 100% free memory [01:40:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.27, 6.62, 5.46 [01:41:22] PROBLEM Total processes is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:42:02] PROBLEM dpkg-check is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:42:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [01:46:23] RECOVERY Total processes is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: PROCS OK: 107 processes [01:47:03] RECOVERY dpkg-check is now: OK on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: All packages OK [01:49:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.31, 8.26, 6.15 [01:58:53] PROBLEM Current Load is now: CRITICAL on tools-login.pmtpa.wmflabs 10.4.0.220 output: Connection refused by host [01:59:33] PROBLEM Disk Space is now: CRITICAL on tools-login.pmtpa.wmflabs 10.4.0.220 output: Connection refused by host [02:00:12] PROBLEM Free ram is now: CRITICAL on tools-login.pmtpa.wmflabs 10.4.0.220 output: Connection refused by host [02:02:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 9.10, 8.71, 6.59 [02:03:52] RECOVERY Current Load is now: OK on tools-login.pmtpa.wmflabs 10.4.0.220 output: OK - load average: 0.20, 0.77, 0.56 [02:04:32] RECOVERY Disk Space is now: OK on tools-login.pmtpa.wmflabs 10.4.0.220 output: DISK OK [02:05:13] RECOVERY Free ram is now: OK on tools-login.pmtpa.wmflabs 10.4.0.220 output: OK: 91% free memory [02:12:54] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [02:14:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 193 processes [02:42:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [02:49:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [03:12:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [03:42:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [03:45:33] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [03:47:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 1.29, 1.91, 5.00 [03:50:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:05:23] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.20, 2.29, 4.34 [04:09:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 2.29, 2.51, 4.49 [04:12:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [04:30:42] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 93% free memory [04:37:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [04:39:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [04:40:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [04:42:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [04:53:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [05:00:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [05:02:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [05:12:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [05:42:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [06:12:52] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [06:27:15] PROBLEM Total processes is now: CRITICAL on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS CRITICAL: 204 processes [06:32:52] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:42:53] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [06:57:53] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 147 processes [06:59:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 195 processes [07:04:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [07:09:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 181 processes [07:13:02] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [07:43:02] PROBLEM host: bots-cb.pmtpa.wmflabs is DOWN address: 10.4.0.44 CRITICAL - Host Unreachable (10.4.0.44) [08:12:12] RECOVERY host: bots-cb.pmtpa.wmflabs is UP address: 10.4.0.44 PING OK - Packet loss = 0%, RTA = 0.55 ms [08:15:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 88% free memory [08:37:13] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [08:38:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [08:40:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [08:49:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [08:53:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [09:00:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [09:04:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 197 processes [09:11:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [09:47:42] PROBLEM Disk Space is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: DISK WARNING - free space: / 481 MB (5% inode=70%): [10:02:43] RECOVERY Disk Space is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: DISK OK [10:23:52] PROBLEM Current Load is now: CRITICAL on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: Connection refused by host [10:24:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 209 processes [10:24:32] PROBLEM Disk Space is now: CRITICAL on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: Connection refused by host [10:25:13] PROBLEM Free ram is now: CRITICAL on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: Connection refused by host [10:26:43] PROBLEM Total processes is now: CRITICAL on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: Connection refused by host [10:27:23] PROBLEM dpkg-check is now: CRITICAL on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: Connection refused by host [10:29:33] RECOVERY Disk Space is now: OK on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: DISK OK [10:30:12] RECOVERY Free ram is now: OK on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: OK: 89% free memory [10:31:42] RECOVERY Total processes is now: OK on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: PROCS OK: 84 processes [10:32:22] RECOVERY dpkg-check is now: OK on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: All packages OK [10:33:52] RECOVERY Current Load is now: OK on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: OK - load average: 0.02, 0.47, 0.46 [11:04:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [11:05:02] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 363 processes [11:10:02] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 196 processes [11:10:23] PROBLEM dpkg-check is now: CRITICAL on puppety-pupp.pmtpa.wmflabs 10.4.0.232 output: DPKG CRITICAL dpkg reports broken packages [11:28:52] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 557 processes [12:20:42] PROBLEM Current Load is now: WARNING on blamemaps-m1xsmall.pmtpa.wmflabs 10.4.0.142 output: WARNING - load average: 17.20, 24.73, 13.32 [12:29:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 205 processes [12:31:52] PROBLEM Current Load is now: WARNING on resourceloader2-apache.pmtpa.wmflabs 10.4.0.128 output: WARNING - load average: 4.76, 4.99, 5.04 [12:34:12] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [12:36:52] RECOVERY Current Load is now: OK on resourceloader2-apache.pmtpa.wmflabs 10.4.0.128 output: OK - load average: 4.54, 4.71, 4.90 [12:38:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [12:39:12] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 20% free memory [12:40:13] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [12:41:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [12:47:12] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [12:48:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [12:50:43] RECOVERY Current Load is now: OK on blamemaps-m1xsmall.pmtpa.wmflabs 10.4.0.142 output: OK - load average: 3.94, 3.63, 4.85 [12:51:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 15% free memory [12:54:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [13:06:42] PROBLEM dpkg-check is now: CRITICAL on cvn-apache2.pmtpa.wmflabs 10.4.0.204 output: DPKG CRITICAL dpkg reports broken packages [13:11:42] RECOVERY dpkg-check is now: OK on cvn-apache2.pmtpa.wmflabs 10.4.0.204 output: All packages OK [13:14:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 197 processes [13:39:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 244 processes [15:18:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:18:54] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [15:24:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 374 processes [15:49:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:49:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [16:19:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:19:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [16:38:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [16:39:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:41:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [16:49:18] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:49:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 194 processes [16:49:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [16:51:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [16:59:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 19% free memory [17:07:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [17:16:13] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 152 processes [17:19:23] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:19:33] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [17:21:12] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 145 processes [17:49:23] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:49:33] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [18:03:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 21% free memory [18:09:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 203 processes [18:19:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:19:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [18:43:53] PROBLEM Current Load is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: Connection refused by host [18:44:34] PROBLEM Disk Space is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: Connection refused by host [18:45:12] PROBLEM Free ram is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: Connection refused by host [18:46:42] PROBLEM Total processes is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: Connection refused by host [18:47:22] PROBLEM dpkg-check is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: Connection refused by host [18:49:32] RECOVERY Disk Space is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: DISK OK [18:49:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:49:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [18:50:13] RECOVERY Free ram is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: OK: 89% free memory [18:51:43] RECOVERY Total processes is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: PROCS OK: 88 processes [18:52:23] RECOVERY dpkg-check is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: All packages OK [18:53:53] RECOVERY Current Load is now: OK on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: OK - load average: 0.02, 0.45, 0.44 [19:04:13] RECOVERY host: blamemaps-m1xsmall.pmtpa.wmflabs is UP address: 10.4.0.142 PING OK - Packet loss = 0%, RTA = 0.54 ms [19:19:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [19:29:42] RECOVERY host: resourceloader2-apache.pmtpa.wmflabs is UP address: 10.4.0.128 PING OK - Packet loss = 0%, RTA = 0.52 ms [19:59:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [20:15:14] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:15:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 7.04, 6.93, 5.61 [20:18:54] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:19:04] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [20:19:14] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:19:24] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:19:24] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [20:19:34] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:24:12] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 158 processes [20:24:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 187 processes [20:24:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:33:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 5.40, 5.57, 5.08 [20:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [20:39:55] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [20:40:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 2.01, 2.79, 4.62 [20:41:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [20:41:22] PROBLEM Free ram is now: WARNING on sultest2.pmtpa.wmflabs 10.4.0.200 output: Warning: 17% free memory [20:45:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:49:33] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [20:49:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:49:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:49:53] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:50:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:53:23] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.79, 3.48, 4.48 [20:55:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:56:26] RECOVERY Free ram is now: OK on sultest2.pmtpa.wmflabs 10.4.0.200 output: OK: 25% free memory [20:57:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 15% free memory [21:00:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [21:14:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [21:15:52] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [21:19:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [21:20:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [21:20:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [21:20:42] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [21:21:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [21:26:22] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [21:45:56] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [21:50:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [21:50:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [21:50:42] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [21:51:02] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [21:51:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [21:53:32] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 1.08, 13.48, 10.52 [21:57:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [22:08:33] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 0.02, 0.75, 4.12 [22:16:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [22:19:13] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 148 processes [22:20:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [22:20:43] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [22:21:03] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [22:21:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [22:22:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [22:27:22] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [22:47:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [22:50:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [22:50:43] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [22:51:03] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [22:51:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [22:52:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [22:57:22] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [23:14:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 220 processes [23:17:05] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [23:20:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [23:20:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [23:21:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [23:21:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [23:22:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [23:22:12] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [23:23:32] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev.pmtpa.wmflabs 10.4.1.55 output: DPKG CRITICAL dpkg reports broken packages [23:27:23] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [23:28:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 9.99, 8.88, 6.42 [23:28:53] PROBLEM Current Load is now: CRITICAL on openstack-role-dev2.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [23:29:33] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev2.pmtpa.wmflabs 10.4.0.81 output: Connection refused by host [23:31:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.81, 8.40, 6.16 [23:32:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.33, 9.82, 6.91 [23:33:52] RECOVERY Current Load is now: OK on openstack-role-dev2.pmtpa.wmflabs 10.4.0.81 output: OK - load average: 0.37, 0.86, 0.60 [23:34:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 186 processes [23:34:32] RECOVERY Disk Space is now: OK on openstack-role-dev2.pmtpa.wmflabs 10.4.0.81 output: DISK OK [23:38:52] PROBLEM Current Load is now: CRITICAL on tools-master.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [23:43:22] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev2.pmtpa.wmflabs 10.4.0.81 output: DPKG CRITICAL dpkg reports broken packages [23:43:52] RECOVERY Current Load is now: OK on tools-master.pmtpa.wmflabs 10.4.0.246 output: OK - load average: 0.02, 0.47, 0.46 [23:47:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [23:50:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [23:50:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [23:51:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [23:51:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [23:52:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [23:53:22] RECOVERY dpkg-check is now: OK on openstack-role-dev2.pmtpa.wmflabs 10.4.0.81 output: All packages OK [23:53:52] PROBLEM Current Load is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [23:53:52] PROBLEM Disk Space is now: CRITICAL on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: Connection refused by host [23:53:52] PROBLEM Free ram is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [23:54:32] PROBLEM Disk Space is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [23:54:32] PROBLEM Free ram is now: CRITICAL on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: Connection refused by host [23:55:13] PROBLEM Free ram is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [23:55:13] PROBLEM Total processes is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [23:56:03] PROBLEM Total processes is now: CRITICAL on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: Connection refused by host [23:56:03] PROBLEM dpkg-check is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [23:56:43] PROBLEM Total processes is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [23:56:43] PROBLEM dpkg-check is now: CRITICAL on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: Connection refused by host [23:57:23] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [23:57:23] PROBLEM dpkg-check is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [23:57:23] PROBLEM Current Load is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [23:57:53] PROBLEM Current Load is now: CRITICAL on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: Connection refused by host [23:58:03] PROBLEM Disk Space is now: CRITICAL on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: Connection refused by host [23:59:33] RECOVERY Disk Space is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: DISK OK