[00:02:53] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 235 processes [00:05:53] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 230 processes [00:07:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:08:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:12:19] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [00:15:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [00:17:13] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [00:18:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:35:23] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 86% free memory [00:37:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:38:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:39:33] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [00:40:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 18% free memory [00:45:23] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [00:48:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:53:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [00:56:03] PROBLEM Free ram is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:01:03] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 9% free memory [01:08:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:08:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:12:03] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 289 processes [01:18:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:20:53] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 197 processes [01:22:03] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 104 processes [01:24:03] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 9.31, 10.46, 7.20 [01:38:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:43:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 157 processes [01:44:03] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.20, 3.45, 4.60 [01:48:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:57:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [02:08:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:21:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:38:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:51:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:08:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:22:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:35:53] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [03:38:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:52:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:01:53] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 156 processes [04:08:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:22:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:33:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 170 processes [04:36:53] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [04:38:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:38:45] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:38:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 107 processes [04:43:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [04:52:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:08:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:22:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:38:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:52:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:06:03] RECOVERY Free ram is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK: 26% free memory [06:08:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:22:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:26:55] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 161 processes [06:31:52] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 157 processes [06:31:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 110 processes [06:31:52] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 156 processes [06:38:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:52:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:56:03] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 368 MB (3% inode=81%): [06:56:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [06:57:53] PROBLEM Total processes is now: WARNING on deployment-squid.pmtpa.wmflabs 10.4.0.17 output: PROCS WARNING: 187 processes [07:08:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:22:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:38:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:38:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:53:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:08:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:08:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:23:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:39:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:39:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:45:53] PROBLEM Current Load is now: CRITICAL on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [08:46:33] PROBLEM Disk Space is now: CRITICAL on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [08:47:13] PROBLEM Free ram is now: CRITICAL on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [08:48:43] PROBLEM Total processes is now: CRITICAL on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [08:49:23] PROBLEM dpkg-check is now: CRITICAL on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [08:53:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:06:53] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [09:09:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:09:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:12:12] PROBLEM Free ram is now: UNKNOWN on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: NRPE: Unable to read output [09:13:42] RECOVERY Total processes is now: OK on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: PROCS OK: 74 processes [09:15:53] RECOVERY Current Load is now: OK on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: OK - load average: 0.02, 0.08, 0.14 [09:16:33] RECOVERY Disk Space is now: OK on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: DISK OK [09:19:33] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 28% free memory [09:23:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:39:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:41:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:53:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:06:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 163 processes [10:09:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:11:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:14:22] RECOVERY dpkg-check is now: OK on deployment-nginx-test.pmtpa.wmflabs 10.4.0.42 output: All packages OK [10:16:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 209 processes [10:21:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 159 processes [10:23:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:39:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:42:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:53:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:12:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:13:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:23:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:42:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:44:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:53:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:12:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:14:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:23:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:24:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 169 processes [12:32:52] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 295 processes [12:34:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 116 processes [12:36:53] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 341 processes [12:41:02] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 21% free memory [12:42:22] PROBLEM Free ram is now: CRITICAL on rds.pmtpa.wmflabs 10.4.0.18 output: Critical: 4% free memory [12:43:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:44:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:47:13] PROBLEM Free ram is now: WARNING on rds.pmtpa.wmflabs 10.4.0.18 output: Warning: 6% free memory [12:53:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:13:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:14:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:24:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:42:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 155 processes [13:43:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:44:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:55:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:12:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [14:13:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:14:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:16:52] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 197 processes [14:26:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:43:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:44:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:57:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:13:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:14:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:27:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:43:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:44:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:57:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:10:13] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 9% free memory [16:13:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:14:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:20:53] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 3% free memory [16:28:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:30:53] PROBLEM Current Load is now: CRITICAL on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:31:33] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:32:13] PROBLEM Free ram is now: CRITICAL on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:33:43] PROBLEM Total processes is now: CRITICAL on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:34:23] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Error - Could not complete SSL handshake. [16:36:53] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 156 processes [16:42:13] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.1.55 output: PROCS WARNING: 161 processes [16:43:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:44:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:45:53] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 26% free memory [16:48:43] RECOVERY Total processes is now: OK on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: PROCS OK: 74 processes [16:50:52] RECOVERY Current Load is now: OK on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.02, 0.08, 0.13 [16:51:32] RECOVERY Disk Space is now: OK on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: DISK OK [16:52:12] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.1.55 output: PROCS OK: 131 processes [16:52:12] PROBLEM Free ram is now: UNKNOWN on openstack-role-dev5.pmtpa.wmflabs 10.4.0.115 output: NRPE: Unable to read output [16:55:02] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [16:58:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:13:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:14:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:25:52] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 8% free memory [17:26:52] RECOVERY Total processes is now: OK on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS OK: 148 processes [17:28:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:44:35] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:44:52] PROBLEM Total processes is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: PROCS WARNING: 151 processes [17:45:53] PROBLEM Free ram is now: CRITICAL on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Critical: 5% free memory [17:46:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:58:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:00:56] PROBLEM Free ram is now: WARNING on ee-prototype.pmtpa.wmflabs 10.4.0.93 output: Warning: 13% free memory [18:14:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:17:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:24:02] PROBLEM Free ram is now: WARNING on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: Warning: 19% free memory [18:28:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:30:23] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.1.55 output: PROCS WARNING: 166 processes [18:35:23] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.1.55 output: PROCS OK: 136 processes [18:44:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:48:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:58:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:15:53] PROBLEM Total processes is now: CRITICAL on unicorn.pmtpa.wmflabs 10.4.0.192 output: Connection refused by host [19:16:33] PROBLEM dpkg-check is now: CRITICAL on unicorn.pmtpa.wmflabs 10.4.0.192 output: Connection refused by host [19:16:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:17:23] PROBLEM Free ram is now: CRITICAL on unicorn.pmtpa.wmflabs 10.4.0.192 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:18:03] PROBLEM Current Load is now: CRITICAL on unicorn.pmtpa.wmflabs 10.4.0.192 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:18:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:18:43] PROBLEM Disk Space is now: CRITICAL on unicorn.pmtpa.wmflabs 10.4.0.192 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:23:03] RECOVERY Current Load is now: OK on unicorn.pmtpa.wmflabs 10.4.0.192 output: OK - load average: 0.12, 0.47, 0.47 [19:23:43] RECOVERY Disk Space is now: OK on unicorn.pmtpa.wmflabs 10.4.0.192 output: DISK OK [19:25:53] RECOVERY Total processes is now: OK on unicorn.pmtpa.wmflabs 10.4.0.192 output: PROCS OK: 81 processes [19:27:23] PROBLEM Free ram is now: UNKNOWN on unicorn.pmtpa.wmflabs 10.4.0.192 output: NRPE: Unable to read output [19:28:20] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:33:52] PROBLEM Current Load is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:34:32] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:35:13] PROBLEM Free ram is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:36:23] PROBLEM Total processes is now: WARNING on wsexport.pmtpa.wmflabs 10.4.0.231 output: PROCS WARNING: 165 processes [19:36:43] PROBLEM Total processes is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:37:23] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:39:33] RECOVERY Disk Space is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: DISK OK [19:40:13] PROBLEM Free ram is now: UNKNOWN on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: NRPE: Unable to read output [19:41:23] RECOVERY Total processes is now: OK on wsexport.pmtpa.wmflabs 10.4.0.231 output: PROCS OK: 94 processes [19:41:43] RECOVERY Total processes is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: PROCS OK: 85 processes [19:43:53] RECOVERY Current Load is now: OK on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: OK - load average: 0.00, 0.15, 0.30 [19:46:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:48:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:59:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:02:02] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 320 processes [20:12:02] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 188 processes [20:15:23] PROBLEM Free ram is now: CRITICAL on openstack-role-dev6.pmtpa.wmflabs 10.4.0.157 output: CHECK_NRPE: Socket timeout after 10 seconds. [20:16:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:16:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 150 processes [20:18:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:19:53] PROBLEM host: openstack-role-dev6.pmtpa.wmflabs is DOWN address: 10.4.0.157 CRITICAL - Host Unreachable (10.4.0.157) [20:25:53] PROBLEM Current Load is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:26:33] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:26:43] PROBLEM Total processes is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [20:27:23] PROBLEM Free ram is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:27:33] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:29:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:45:53] RECOVERY Current Load is now: OK on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.01, 0.07, 0.16 [20:46:33] RECOVERY Disk Space is now: OK on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: DISK OK [20:46:43] RECOVERY Total processes is now: OK on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: PROCS OK: 79 processes [20:46:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:47:23] PROBLEM Free ram is now: UNKNOWN on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: NRPE: Unable to read output [20:48:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:59:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:15:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [21:16:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:18:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:22:32] RECOVERY dpkg-check is now: OK on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: All packages OK [21:29:07] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:41:43] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.0.115 output: DPKG CRITICAL dpkg reports broken packages [21:46:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:48:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:59:03] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:16:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:18:43] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:31:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:47:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:48:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:01:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:05:13] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 187 processes [23:10:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 114 processes [23:17:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:18:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:31:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:47:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:48:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253)