[00:01:47] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:01:53] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 248 processes [00:03:03] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 232 processes [00:12:33] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [00:26:03] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:31:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:38:03] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 154 processes [00:48:07] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:48:57] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:01:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:10:23] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 239 processes [01:13:03] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [01:18:03] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 150 processes [01:18:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:19:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:20:23] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 124 processes [01:25:53] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 8% free memory [01:27:13] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 3.49, 5.92, 5.08 [01:30:53] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 4% free memory [01:31:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:32:13] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.38, 4.29, 4.60 [01:37:03] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [01:39:43] PROBLEM Free ram is now: WARNING on bots-4.pmtpa.wmflabs 10.4.0.64 output: Warning: 14% free memory [01:50:25] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:52:45] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:55:14] PROBLEM Free ram is now: CRITICAL on bots-4.pmtpa.wmflabs 10.4.0.64 output: Critical: 4% free memory [02:00:13] RECOVERY Free ram is now: OK on bots-4.pmtpa.wmflabs 10.4.0.64 output: OK: 53% free memory [02:01:13] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 157 processes [02:03:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:20:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:23:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:29:23] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 7% free memory [02:33:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:41:43] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [02:50:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:53:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:03:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:20:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:23:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:33:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:44:43] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 5% free memory [03:50:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:53:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:59:43] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 6% free memory [04:03:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:09:33] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [04:14:42] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 4% free memory [04:20:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:23:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:26:22] PROBLEM Total processes is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:31:13] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 150 processes [04:33:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:39:13] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 152 processes [04:53:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:54:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:59:42] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 6% free memory [05:03:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:23:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:24:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:34:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:41:33] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 8% free memory [05:53:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:54:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:04:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:17:13] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 176 processes [06:17:23] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 151 processes [06:23:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:24:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:27:15] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 119 processes [06:32:23] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 150 processes [06:32:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 156 processes [06:34:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:53:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:54:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:57:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [07:05:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:23:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:24:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:35:33] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 151 processes [07:35:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:36:33] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 5% free memory [07:53:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:54:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:56:33] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 8% free memory [08:05:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:16:43] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 4% free memory [08:24:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:24:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:30:32] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 149 processes [08:35:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:38:33] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 153 processes [08:54:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:55:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:05:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:21:43] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 3% free memory [09:25:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:26:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:28:33] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 145 processes [09:35:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:36:53] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 25% free memory [09:41:43] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 10% free memory [09:55:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:56:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:59:33] PROBLEM Disk Space is now: CRITICAL on zarro.pmtpa.wmflabs 10.4.0.157 output: Connection refused by host [10:00:23] PROBLEM Free ram is now: CRITICAL on zarro.pmtpa.wmflabs 10.4.0.157 output: Connection refused by host [10:00:53] PROBLEM Current Load is now: CRITICAL on zarro.pmtpa.wmflabs 10.4.0.157 output: Connection refused by host [10:01:43] PROBLEM Total processes is now: CRITICAL on zarro.pmtpa.wmflabs 10.4.0.157 output: Connection refused by host [10:02:33] PROBLEM dpkg-check is now: CRITICAL on zarro.pmtpa.wmflabs 10.4.0.157 output: Connection refused by host [10:05:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:25:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:26:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:32:13] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 258 processes [10:35:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:47:13] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 122 processes [10:50:13] PROBLEM Free ram is now: UNKNOWN on boogs.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [10:55:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:56:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:06:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:25:13] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 245 processes [11:25:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:26:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:37:37] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:55:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:56:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:08:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:11:43] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 5% free memory [12:16:32] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 158 processes [12:25:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:26:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:38:45] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:56:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:57:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:00:12] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 8% free memory [13:08:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:10:13] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 5% free memory [13:26:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:27:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:39:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:55:13] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 6% free memory [13:56:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:57:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:09:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:25:12] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 3% free memory [14:26:02] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 149 processes [14:26:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:28:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:39:25] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:46:25] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 6% free memory [14:56:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:58:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:09:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:26:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:28:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:40:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:56:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:59:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:00:13] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [16:10:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:15:13] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 20% free memory [16:16:03] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [16:17:23] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 6% free memory [16:19:23] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [16:27:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:28:22] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [16:29:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:40:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:57:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:01:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:10:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:12:23] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 151 processes [17:27:23] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 148 processes [17:27:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:31:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:41:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:57:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:02:32] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 151 processes [18:02:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:11:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:27:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:32:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:42:14] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:54:15] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 152 processes [18:58:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:59:03] PROBLEM Current Load is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [18:59:43] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [19:00:23] PROBLEM Free ram is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:02:33] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: DPKG CRITICAL dpkg reports broken packages [19:02:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:04:03] RECOVERY Current Load is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.57, 0.66, 0.39 [19:04:23] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [19:04:43] RECOVERY Disk Space is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: DISK OK [19:05:33] PROBLEM Free ram is now: UNKNOWN on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: NRPE: Unable to read output [19:07:33] RECOVERY dpkg-check is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: All packages OK [19:11:42] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [19:13:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:15:43] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.115 output: DPKG CRITICAL dpkg reports broken packages [19:16:45] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 20% free memory [19:28:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:33:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:39:43] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [19:43:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:58:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:03:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:13:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:28:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:33:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:43:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:43:53] PROBLEM Current Load is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: Connection refused by host [20:44:34] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:45:24] PROBLEM Free ram is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:46:44] PROBLEM Total processes is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:47:24] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:53:22] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 151 processes [20:57:13] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: CHECK_NRPE: Socket timeout after 10 seconds. [20:58:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:01:53] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [21:03:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:13:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:14:03] RECOVERY Current Load is now: OK on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: OK - load average: 0.07, 0.06, 0.07 [21:14:53] RECOVERY Disk Space is now: OK on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: DISK OK [21:15:32] PROBLEM Free ram is now: UNKNOWN on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: NRPE: Unable to read output [21:17:22] RECOVERY Total processes is now: OK on openstack-role-dev7.pmtpa.wmflabs 10.4.0.157 output: PROCS OK: 79 processes [21:29:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:33:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 270 processes [21:33:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:36:33] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 152 processes [21:38:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 123 processes [21:43:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:56:32] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 149 processes [21:58:22] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [21:59:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:03:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:08:23] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 20% free memory [22:13:33] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [22:13:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:18:52] PROBLEM Current Load is now: CRITICAL on tools-exec-03.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:19:32] PROBLEM Disk Space is now: CRITICAL on tools-exec-03.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:20:13] PROBLEM Free ram is now: CRITICAL on tools-exec-03.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:22:23] PROBLEM dpkg-check is now: CRITICAL on tools-exec-03.pmtpa.wmflabs 10.4.0.115 output: DPKG CRITICAL dpkg reports broken packages [22:23:53] RECOVERY Current Load is now: OK on tools-exec-03.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.04, 0.20, 0.20 [22:24:33] RECOVERY Disk Space is now: OK on tools-exec-03.pmtpa.wmflabs 10.4.0.115 output: DISK OK [22:25:13] PROBLEM Free ram is now: UNKNOWN on tools-exec-03.pmtpa.wmflabs 10.4.0.115 output: NRPE: Unable to read output [22:29:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:31:23] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [22:33:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:44:03] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [22:45:55] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:59:17] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253)