[00:09:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:22:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:29:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:35:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 4% free memory [00:39:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:40:24] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 12% free memory [00:52:34] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:59:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:09:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:11:03] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 334 processes [01:16:03] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 100 processes [01:20:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 3% free memory [01:20:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 160 processes [01:22:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:25:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 109 processes [01:29:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:39:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:46:33] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 27% free memory [01:52:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:59:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:09:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:24:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:29:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:35:22] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 11% free memory [02:39:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:54:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:59:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:09:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:24:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:29:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:39:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:54:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:59:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:09:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:24:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:31:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:39:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:50:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [04:54:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:55:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 6% free memory [05:00:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 4% free memory [05:02:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:02:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Critical: 5% free memory [05:05:12] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 182 processes [05:06:02] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: CHECK_NRPE: Socket timeout after 10 seconds. [05:07:52] PROBLEM Free ram is now: WARNING on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Warning: 6% free memory [05:10:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 179 processes [05:13:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:17:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Critical: 5% free memory [05:24:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:32:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:43:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:54:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:55:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 10% free memory [06:02:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:13:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:24:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:28:45] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 156 processes [06:29:55] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 153 processes [06:31:53] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 157 processes [06:32:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:43:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:46:52] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [06:50:22] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [06:53:42] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [06:54:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:54:52] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [07:02:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:13:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:24:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:27:03] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [07:32:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:43:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:54:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:02:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:04:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 154 processes [08:09:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 138 processes [08:13:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:15:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 7% free memory [08:24:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:25:23] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [08:32:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:43:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:56:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:58:23] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [09:00:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 10% free memory [09:02:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:14:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:26:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:30:53] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 156 processes [09:39:36] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:47:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:47:22] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 1% free memory [09:47:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 176 processes [09:56:19] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:03:19] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:03:19] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 128 processes [10:05:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 8% free memory [10:10:22] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [10:14:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:27:07] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 232 processes [10:27:08] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 311 processes [10:27:09] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:33:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:43:48] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 178 processes [10:44:38] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:56:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:58:23] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [11:03:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:14:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:26:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:26:23] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [11:33:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:46:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:50:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [11:56:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:00:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 6% free memory [12:03:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:17:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:27:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:33:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:48:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:57:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:03:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:10:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [13:15:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 6% free memory [13:17:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [13:18:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:20:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [13:26:23] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [13:26:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 182 processes [13:27:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:34:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:48:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:54:33] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [13:55:53] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [13:56:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 142 processes [13:57:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:00:53] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 203 processes [14:04:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:05:52] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [14:18:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:27:32] PROBLEM Disk Space is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:27:52] PROBLEM Free ram is now: UNKNOWN on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: NRPE: Call to fork() failed [14:29:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:30:33] PROBLEM Current Load is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:30:53] PROBLEM Total processes is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:31:03] PROBLEM SSH is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Server answer: [14:31:13] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:32:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: CHECK_NRPE: Error - Could not complete SSL handshake. [14:34:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:35:34] RECOVERY Current Load is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK - load average: 1.10, 1.11, 0.97 [14:35:54] RECOVERY Total processes is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: PROCS OK: 118 processes [14:41:03] RECOVERY SSH is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:41:13] RECOVERY dpkg-check is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: All packages OK [14:42:33] RECOVERY Disk Space is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: DISK OK [14:42:53] RECOVERY Free ram is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK: 21% free memory [14:48:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:00:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:04:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:19:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:30:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:34:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:40:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 11% free memory [15:43:13] RECOVERY Free ram is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: OK: 96% free memory [15:43:53] RECOVERY Current Load is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: OK - load average: 0.19, 0.15, 0.06 [15:44:23] RECOVERY dpkg-check is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: All packages OK [15:44:33] RECOVERY Total processes is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: PROCS OK: 87 processes [15:44:43] RECOVERY Current Load is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: OK - load average: 0.74, 0.32, 0.12 [15:44:43] RECOVERY Current Load is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: OK - load average: 0.36, 0.14, 0.05 [15:45:22] RECOVERY Disk Space is now: OK on cephticon1.pmtpa.wmflabs 10.4.1.56 output: DISK OK [15:45:42] RECOVERY Disk Space is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: DISK OK [15:46:42] RECOVERY Disk Space is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: DISK OK [15:47:02] RECOVERY Total processes is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: PROCS OK: 81 processes [15:47:02] RECOVERY Free ram is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: OK: 96% free memory [15:48:02] RECOVERY Free ram is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: OK: 95% free memory [15:48:22] RECOVERY Total processes is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: PROCS OK: 87 processes [15:48:32] RECOVERY dpkg-check is now: OK on cephticon3.pmtpa.wmflabs 10.4.1.72 output: All packages OK [15:48:42] RECOVERY dpkg-check is now: OK on cephticon2.pmtpa.wmflabs 10.4.0.249 output: All packages OK [15:49:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:55:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 4% free memory [15:59:33] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [16:00:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:04:56] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:19:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:26:54] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 169 processes [16:27:34] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [16:29:34] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [16:31:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:35:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:35:33] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 1% free memory [16:49:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:51:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 132 processes [16:55:53] PROBLEM Free ram is now: WARNING on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Warning: 19% free memory [17:01:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:06:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:19:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:31:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:37:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:49:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:01:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:07:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:23:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:27:33] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [18:31:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:35:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 7% free memory [18:37:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:53:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:55:33] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [19:01:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:07:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:15:54] PROBLEM Current Load is now: CRITICAL on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: Connection refused by host [19:16:33] PROBLEM Disk Space is now: CRITICAL on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: Connection refused by host [19:16:43] PROBLEM Total processes is now: CRITICAL on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: Connection refused by host [19:17:13] PROBLEM Free ram is now: CRITICAL on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: Connection refused by host [19:17:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: Connection refused by host [19:20:52] RECOVERY Current Load is now: OK on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: OK - load average: 0.99, 0.79, 0.44 [19:21:32] RECOVERY Disk Space is now: OK on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: DISK OK [19:21:42] RECOVERY Total processes is now: OK on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: PROCS OK: 90 processes [19:22:12] PROBLEM Free ram is now: UNKNOWN on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: NRPE: Unable to read output [19:22:22] RECOVERY dpkg-check is now: OK on mwreview-test1.pmtpa.wmflabs 10.4.0.32 output: All packages OK [19:23:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:31:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:37:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:53:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:01:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:07:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:10:22] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [20:15:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 8% free memory [20:24:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:31:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:37:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:54:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:55:33] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [21:01:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:07:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:24:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:28:33] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [21:32:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:37:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:54:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:02:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:07:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:24:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:32:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:37:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:54:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:02:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:07:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:24:39] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:28:43] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [23:32:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:37:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:54:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:56:42] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory