[00:03:08] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:10:19] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:19:55] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:25:41] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 164 processes [00:30:41] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 133 processes [00:33:11] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:40:51] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:43:51] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 164 processes [00:50:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:03:30] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:10:13] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 228 processes [01:10:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:18:03] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 95 processes [01:22:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:33:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:40:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:52:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:03:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:10:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:23:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:33:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:42:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:53:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:03:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:12:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:23:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:33:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:42:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:53:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:03:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:12:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:23:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:33:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:42:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:53:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:04:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:12:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:24:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:34:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:40:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [05:42:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:45:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 9% free memory [05:54:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:04:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:14:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:24:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:30:32] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 156 processes [06:31:52] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 153 processes [06:34:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:44:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:46:53] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [06:54:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:55:22] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [06:55:32] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [07:04:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:14:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:15:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 6% free memory [07:24:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:34:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:44:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:54:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:04:28] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:14:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:26:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:34:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:46:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:57:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:02:12] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 0.18, 1.26, 3.86 [09:04:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:16:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:27:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:34:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:46:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:57:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:04:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:16:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:36:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:46:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:07:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:16:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:38:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:47:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:08:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:17:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:38:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:47:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:00:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 4% free memory [13:05:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 8% free memory [13:08:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:17:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:20:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 3% free memory [13:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:28:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 182 processes [13:38:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:47:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:08:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 142 processes [14:17:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:20:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 9% free memory [14:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:47:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:17:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:49:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:15:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [16:19:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:21:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 158 processes [16:25:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 10% free memory [16:26:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 97 processes [16:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:45:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [16:49:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:19:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:49:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:19:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:49:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:57:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:05:23] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 3% free memory [19:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:11:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 204 processes [19:16:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 188 processes [19:19:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:21:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 211 processes [19:27:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:45:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 8% free memory [19:49:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:57:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:15:32] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 2% free memory [20:20:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:28:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:30:22] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 9% free memory [20:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:51:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:58:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:21:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:28:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:51:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:59:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:21:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:29:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:51:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:59:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:09:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:21:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:29:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:30:22] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 11% free memory [23:36:33] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [23:39:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:51:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:58:03] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [23:59:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26)