[00:18:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:27:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:29:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:48:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:59:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:59:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:05:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [01:10:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [01:12:53] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 344 processes [01:21:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:22:53] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 110 processes [01:27:53] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 8.20, 9.49, 6.94 [01:29:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:30:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:42:53] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.56, 3.50, 4.59 [01:51:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 19% free memory [01:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:59:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:01:42] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [02:02:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:20:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [02:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:31:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:32:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:40:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [02:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:01:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:02:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:31:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:32:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:00:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [04:02:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:03:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:04:02] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 129 processes [04:05:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 16% free memory [04:05:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:11:03] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 203 processes [04:22:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:30:44] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [04:32:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:33:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:52:26] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:00:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [05:02:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:04:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:04:42] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 19% free memory [05:22:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:30:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 8% free memory [05:33:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:34:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:52:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:04:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:04:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:05:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:22:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:29:34] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 153 processes [06:30:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [06:30:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 156 processes [06:34:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:34:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 12% free memory [06:35:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:44:42] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [06:45:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [06:52:24] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:04:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:05:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:05:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:22:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:34:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:35:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:52:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:04:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:05:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:20:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [08:22:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:34:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:37:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:53:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:04:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:05:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [09:08:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:20:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [09:23:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:34:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:34:43] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [09:39:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:53:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:06:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:09:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:13:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 17% free memory [10:24:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:36:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:38:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 175 processes [10:39:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:48:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 145 processes [10:55:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:01:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 183 processes [11:06:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:09:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:26:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:36:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:39:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:57:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:00:52] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [12:05:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [12:06:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:09:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:23:42] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [12:23:52] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 154 processes [12:27:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:31:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 19% free memory [12:37:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:39:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:57:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:10:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:27:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:40:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:46:43] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [13:50:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [13:55:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [13:57:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:09:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 18% free memory [14:12:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:27:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:33:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [14:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:42:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:42:52] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 219 processes [14:47:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 177 processes [14:52:56] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 205 processes [14:57:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:09:43] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [15:13:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:27:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:32:42] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 18% free memory [15:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:43:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:44:23] RECOVERY dpkg-check is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.0.27 output: All packages OK [15:45:53] RECOVERY Current Load is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.0.27 output: OK - load average: 0.41, 0.49, 0.27 [15:46:33] RECOVERY Disk Space is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.0.27 output: DISK OK [15:47:13] PROBLEM Free ram is now: UNKNOWN on mwreview-dev5.pmtpa.wmflabs 10.4.0.27 output: NRPE: Unable to read output [15:48:43] RECOVERY Total processes is now: OK on mwreview-dev5.pmtpa.wmflabs 10.4.0.27 output: PROCS OK: 78 processes [15:50:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev6.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [15:51:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev6.pmtpa.wmflabs 10.4.1.105 output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:52:13] PROBLEM Free ram is now: UNKNOWN on mwreview-dev6.pmtpa.wmflabs 10.4.1.105 output: NRPE: Unable to read output [15:55:52] RECOVERY Current Load is now: OK on mwreview-dev6.pmtpa.wmflabs 10.4.1.105 output: OK - load average: 0.06, 0.42, 0.29 [15:56:32] RECOVERY Disk Space is now: OK on mwreview-dev6.pmtpa.wmflabs 10.4.1.105 output: DISK OK [15:57:42] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [15:57:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:00:13] PROBLEM Free ram is now: UNKNOWN on wikiversity-sandbox-world.pmtpa.wmflabs 10.4.1.113 output: NRPE: Unable to read output [16:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:13:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:25:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [16:27:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:30:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [16:35:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 19% free memory [16:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:43:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:57:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:14:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:16:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [17:27:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:34:53] PROBLEM host: mwreview-dev8.pmtpa.wmflabs is DOWN address: 10.4.0.115 CRITICAL - Host Unreachable (10.4.0.115) [17:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:38:13] RECOVERY host: mwreview-dev8.pmtpa.wmflabs is UP address: 10.4.0.115 PING OK - Packet loss = 0%, RTA = 0.73 ms [17:38:43] PROBLEM Total processes is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [17:39:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [17:40:52] PROBLEM Current Load is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [17:41:32] PROBLEM Disk Space is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [17:42:12] PROBLEM Free ram is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [17:43:42] RECOVERY Total processes is now: OK on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: PROCS OK: 76 processes [17:44:22] RECOVERY dpkg-check is now: OK on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: All packages OK [17:44:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:45:53] RECOVERY Current Load is now: OK on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.06, 0.51, 0.35 [17:46:33] RECOVERY Disk Space is now: OK on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: DISK OK [17:47:13] PROBLEM Free ram is now: UNKNOWN on mwreview-dev8.pmtpa.wmflabs 10.4.0.115 output: NRPE: Unable to read output [17:58:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:03:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 170 processes [18:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:13:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 110 processes [18:14:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:28:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:44:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:45:43] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 2% free memory [18:46:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 204 processes [18:50:43] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 8% free memory [19:01:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:05:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [19:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:10:43] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 4% free memory [19:10:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [19:14:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:32:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:37:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:44:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:45:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [19:49:43] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:50:13] PROBLEM Current Load is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:50:53] RECOVERY Current Load is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK - load average: 0.81, 1.09, 0.99 [19:54:43] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [19:55:42] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 14% free memory [20:02:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:08:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:14:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:25:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 8% free memory [20:32:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:39:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:44:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:46:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [20:55:25] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 153 processes [21:02:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:09:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:10:52] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [21:14:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:25:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [21:32:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:39:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:45:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:50:22] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 150 processes [21:55:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [22:02:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:11:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:15:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:30:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 8% free memory [22:32:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:41:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:45:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:02:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:12:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:15:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:25:53] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:32:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:40:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [23:42:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:45:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26)