[00:00:53] RECOVERY Current Load is now: OK on openstack-role-dev4.pmtpa.wmflabs 10.4.1.118 output: OK - load average: 1.20, 0.77, 0.40 [00:01:33] RECOVERY Disk Space is now: OK on openstack-role-dev4.pmtpa.wmflabs 10.4.1.118 output: DISK OK [00:01:43] RECOVERY Total processes is now: OK on openstack-role-dev4.pmtpa.wmflabs 10.4.1.118 output: PROCS OK: 89 processes [00:12:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:25:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:27:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:43:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:48:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [00:57:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:58:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:13:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:27:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:28:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:38:53] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 325 processes [01:43:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:43:53] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 112 processes [01:50:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [01:57:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:58:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:00:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [02:13:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:27:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:28:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:43:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:57:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:58:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:13:15] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:25:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [03:27:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:28:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:30:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [03:43:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:57:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:58:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:13:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:25:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [04:28:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:29:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:46:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:55:54] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:59:34] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:59:44] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:17:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:29:43] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [05:29:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:30:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:31:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 167 processes [05:34:42] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 11% free memory [05:36:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 122 processes [05:47:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:00:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:03:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:17:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:28:45] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 153 processes [06:30:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:31:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 156 processes [06:33:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:38:43] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [06:46:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [06:47:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:50:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [06:51:43] RECOVERY Disk Space is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: DISK OK [06:52:03] RECOVERY Free ram is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: OK: 91% free memory [06:52:03] RECOVERY Total processes is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: PROCS OK: 97 processes [06:52:03] RECOVERY dpkg-check is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: All packages OK [06:52:23] RECOVERY Current Load is now: OK on api1.pmtpa.wmflabs 10.4.1.59 output: OK - load average: 1.48, 0.85, 0.33 [06:55:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [07:00:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:03:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:17:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:30:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:33:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:47:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:00:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:01:33] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 5% free memory [08:03:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:06:33] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [08:17:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:31:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:33:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:47:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:01:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:04:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:18:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:31:42] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 4% free memory [09:32:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:34:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:46:33] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [09:48:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:01:33] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 4% free memory [10:02:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:04:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:18:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:32:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:34:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:46:33] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [10:48:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:05:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:19:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:35:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:50:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:05:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:05:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:10:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [12:21:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:35:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:49:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [12:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:54:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [13:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:07:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:30:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [13:31:33] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 5% free memory [13:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:36:33] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [13:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:43:32] PROBLEM Free ram is now: WARNING on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Warning: 19% free memory [13:45:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:54:23] PROBLEM Disk Space is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: DISK WARNING - free space: / 571 MB (5% inode=75%): [14:01:33] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 4% free memory [14:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:46:33] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [14:50:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [14:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:10:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [15:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:27:52] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 157 processes [15:31:33] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 4% free memory [15:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:37:33] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 155 processes [15:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:46:42] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [15:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:20:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [16:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:35:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [16:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:37:33] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 150 processes [16:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:50:23] PROBLEM Free ram is now: UNKNOWN on mwreview-dev9.pmtpa.wmflabs 10.4.0.27 output: NRPE: Unable to read output [16:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:00:43] PROBLEM Free ram is now: CRITICAL on testing-yp.pmtpa.wmflabs 10.4.0.42 output: CHECK_NRPE: Socket timeout after 10 seconds. [17:05:33] PROBLEM Free ram is now: UNKNOWN on testing-yp.pmtpa.wmflabs 10.4.0.42 output: NRPE: Unable to read output [17:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:11:20] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:33:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 152 processes [17:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:43:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 131 processes [18:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:45:32] PROBLEM Total processes is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS WARNING: 200 processes [18:52:14] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:22:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:28:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev1.pmtpa.wmflabs 10.4.1.103 output: Connection refused by host [19:30:13] PROBLEM Free ram is now: UNKNOWN on mwreview-dev1.pmtpa.wmflabs 10.4.1.103 output: NRPE: Unable to read output [19:31:43] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 3% free memory [19:32:23] PROBLEM Free ram is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Socket timeout after 10 seconds. [19:33:53] RECOVERY Current Load is now: OK on mwreview-dev1.pmtpa.wmflabs 10.4.1.103 output: OK - load average: 0.88, 0.59, 0.35 [19:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:36:43] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [19:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:52:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:52:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [20:04:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 200 processes [20:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:15:02] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 110 processes [20:22:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:40:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 157 processes [20:50:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:51:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:51:43] PROBLEM Total processes is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:52:13] PROBLEM Free ram is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:52:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:52:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:55:53] RECOVERY Current Load is now: OK on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.78, 0.88, 0.45 [20:56:33] RECOVERY Disk Space is now: OK on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: DISK OK [20:56:43] RECOVERY Total processes is now: OK on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: PROCS OK: 77 processes [20:57:13] PROBLEM Free ram is now: UNKNOWN on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: NRPE: Unable to read output [20:57:23] RECOVERY dpkg-check is now: OK on mwreview-dev2.pmtpa.wmflabs 10.4.0.115 output: All packages OK [21:01:42] PROBLEM Free ram is now: CRITICAL on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Critical: 5% free memory [21:04:42] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 1% free memory [21:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:08:03] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 230 processes [21:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:08:53] PROBLEM Free ram is now: UNKNOWN on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: NRPE: Unable to read output [21:09:53] PROBLEM host: solr-mw.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [21:09:53] PROBLEM host: solr1.pmtpa.wmflabs is DOWN address: 10.4.0.246 CRITICAL - Host Unreachable (10.4.0.246) [21:10:03] PROBLEM host: solr2.pmtpa.wmflabs is DOWN address: 10.4.1.21 CRITICAL - Host Unreachable (10.4.1.21) [21:10:53] PROBLEM Disk Space is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:10:53] PROBLEM Total processes is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:11:03] PROBLEM Total processes is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:11:03] PROBLEM Current Load is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:11:33] PROBLEM Free ram is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:11:33] PROBLEM Total processes is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.216 output: Connection refused by host [21:11:33] PROBLEM dpkg-check is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:11:33] PROBLEM Disk Space is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:11:43] PROBLEM dpkg-check is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:11:43] PROBLEM Disk Space is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:12:13] PROBLEM dpkg-check is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.216 output: Connection refused by host [21:12:13] PROBLEM Free ram is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:12:23] PROBLEM Free ram is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:13:13] RECOVERY host: solr-mw.pmtpa.wmflabs is UP address: 10.4.0.250 PING OK - Packet loss = 0%, RTA = 0.62 ms [21:13:43] PROBLEM Total processes is now: CRITICAL on solr-mw.pmtpa.wmflabs 10.4.0.250 output: Connection refused by host [21:13:43] PROBLEM Current Load is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.216 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:13:43] PROBLEM Total processes is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:14:23] PROBLEM dpkg-check is now: CRITICAL on solr-mw.pmtpa.wmflabs 10.4.0.250 output: Connection refused by host [21:14:23] PROBLEM dpkg-check is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:14:33] RECOVERY host: solr1.pmtpa.wmflabs is UP address: 10.4.0.246 PING OK - Packet loss = 0%, RTA = 0.65 ms [21:14:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 14% free memory [21:15:13] PROBLEM Free ram is now: UNKNOWN on solr-zk2.pmtpa.wmflabs 10.4.0.216 output: NRPE: Unable to read output [21:15:13] RECOVERY host: solr2.pmtpa.wmflabs is UP address: 10.4.1.21 PING OK - Packet loss = 0%, RTA = 0.64 ms [21:15:53] RECOVERY Disk Space is now: OK on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: DISK OK [21:15:53] RECOVERY Total processes is now: OK on solr0.pmtpa.wmflabs 10.4.1.42 output: PROCS OK: 91 processes [21:16:03] RECOVERY Total processes is now: OK on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: PROCS OK: 81 processes [21:16:03] RECOVERY Current Load is now: OK on solr0.pmtpa.wmflabs 10.4.1.42 output: OK - load average: 0.05, 0.45, 0.31 [21:16:33] PROBLEM Free ram is now: UNKNOWN on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: NRPE: Unable to read output [21:16:33] RECOVERY Total processes is now: OK on solr-zk2.pmtpa.wmflabs 10.4.0.216 output: PROCS OK: 81 processes [21:16:33] RECOVERY dpkg-check is now: OK on solr0.pmtpa.wmflabs 10.4.1.42 output: All packages OK [21:16:33] PROBLEM Current Load is now: CRITICAL on solr2.pmtpa.wmflabs 10.4.1.21 output: Connection refused by host [21:16:34] RECOVERY Disk Space is now: OK on solr3.pmtpa.wmflabs 10.4.1.46 output: DISK OK [21:16:43] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 10% free memory [21:16:43] RECOVERY dpkg-check is now: OK on solr-zk0.pmtpa.wmflabs 10.4.0.203 output: All packages OK [21:16:43] RECOVERY Disk Space is now: OK on solr0.pmtpa.wmflabs 10.4.1.42 output: DISK OK [21:17:13] PROBLEM Free ram is now: UNKNOWN on solr-mw.pmtpa.wmflabs 10.4.0.250 output: NRPE: Unable to read output [21:17:13] RECOVERY dpkg-check is now: OK on solr-zk2.pmtpa.wmflabs 10.4.0.216 output: All packages OK [21:17:13] PROBLEM Disk Space is now: CRITICAL on solr2.pmtpa.wmflabs 10.4.1.21 output: Connection refused by host [21:17:13] PROBLEM Free ram is now: UNKNOWN on solr3.pmtpa.wmflabs 10.4.1.46 output: NRPE: Unable to read output [21:17:23] PROBLEM Free ram is now: UNKNOWN on solr0.pmtpa.wmflabs 10.4.1.42 output: NRPE: Unable to read output [21:18:03] PROBLEM Free ram is now: UNKNOWN on solr2.pmtpa.wmflabs 10.4.1.21 output: NRPE: Unable to read output [21:18:44] RECOVERY Total processes is now: OK on solr-mw.pmtpa.wmflabs 10.4.0.250 output: PROCS OK: 76 processes [21:18:44] RECOVERY Current Load is now: OK on solr-zk2.pmtpa.wmflabs 10.4.0.216 output: OK - load average: 0.01, 0.28, 0.22 [21:18:44] PROBLEM Free ram is now: UNKNOWN on solr1.pmtpa.wmflabs 10.4.0.246 output: NRPE: Unable to read output [21:18:44] RECOVERY Total processes is now: OK on solr3.pmtpa.wmflabs 10.4.1.46 output: PROCS OK: 76 processes [21:19:23] RECOVERY dpkg-check is now: OK on solr-mw.pmtpa.wmflabs 10.4.0.250 output: All packages OK [21:19:23] RECOVERY dpkg-check is now: OK on solr3.pmtpa.wmflabs 10.4.1.46 output: All packages OK [21:21:33] RECOVERY Current Load is now: OK on solr2.pmtpa.wmflabs 10.4.1.21 output: OK - load average: 0.03, 0.31, 0.22 [21:22:13] RECOVERY Disk Space is now: OK on solr2.pmtpa.wmflabs 10.4.1.21 output: DISK OK [21:22:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:44:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: Connection refused by host [21:45:13] PROBLEM Free ram is now: CRITICAL on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: Connection refused by host [21:49:33] RECOVERY Disk Space is now: OK on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: DISK OK [21:50:13] PROBLEM Free ram is now: UNKNOWN on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: NRPE: Unable to read output [21:52:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:01:03] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 196 processes [22:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:09:43] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 4% free memory [22:22:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:38:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:39:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 16% free memory [22:52:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:06:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:18:33] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 151 processes [23:23:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:24:03] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 185 processes [23:24:43] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [23:36:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:38:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:44:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 17% free memory [23:53:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253)