[00:01:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 290 MB (3% inode=84%): [00:07:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:27:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:31:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 263 MB (2% inode=84%): [00:38:52] PROBLEM Current Load is now: CRITICAL on testing-newimage27.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:39:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:39:32] PROBLEM Disk Space is now: CRITICAL on testing-newimage27.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:40:13] PROBLEM Free ram is now: CRITICAL on testing-newimage27.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:41:43] PROBLEM Total processes is now: CRITICAL on testing-newimage27.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:42:23] PROBLEM dpkg-check is now: CRITICAL on testing-newimage27.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:57:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:03:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:04:32] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 289 MB (3% inode=84%): [01:09:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:16:52] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 359 processes [01:26:53] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 145 processes [01:27:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:29:53] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 5.03, 7.69, 6.20 [01:33:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 266 MB (2% inode=84%): [01:39:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:39:53] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.55, 3.99, 4.83 [01:41:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [01:57:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:03:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 289 MB (3% inode=84%): [02:09:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:27:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:34:32] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 272 MB (2% inode=84%): [02:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:39:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:57:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 289 MB (3% inode=84%): [03:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:09:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:27:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 267 MB (2% inode=84%): [03:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:39:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:57:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 289 MB (3% inode=84%): [04:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:09:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:27:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:34:32] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 276 MB (2% inode=84%): [04:35:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:39:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:57:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 288 MB (3% inode=84%): [05:05:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:09:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:27:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 269 MB (2% inode=84%): [05:36:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:39:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:57:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:04:32] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 288 MB (3% inode=84%): [06:06:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:09:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:27:54] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:30:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 156 processes [06:31:23] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 152 processes [06:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 272 MB (2% inode=84%): [06:36:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:40:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:40:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [06:41:33] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 16% free memory [06:56:22] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 148 processes [06:58:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:06:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:10:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:29:02] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:40:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:00:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:12:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:31:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:42:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:02:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:12:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:31:32] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 23% free memory [09:32:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:42:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:02:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:12:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:32:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:42:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:49:23] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 154 processes [11:02:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:12:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:32:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:34:23] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 149 processes [11:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:42:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:02:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:12:59] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:26:52] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 204 processes [12:29:32] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 16% free memory [12:31:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 97 processes [12:32:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:34:33] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 24% free memory [12:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:42:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:56:13] PROBLEM Current Load is now: WARNING on bots-gs.pmtpa.wmflabs 10.4.1.88 output: WARNING - load average: 10.03, 9.64, 6.89 [13:02:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:14:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:26:13] RECOVERY Current Load is now: OK on bots-gs.pmtpa.wmflabs 10.4.1.88 output: OK - load average: 0.12, 0.76, 3.73 [13:32:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:44:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:02:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:14:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:18:54] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 154 processes [14:32:32] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:44:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:02:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:06:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:14:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:23:53] PROBLEM Current Load is now: CRITICAL on wikiversity-sandbox-web.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [15:24:33] PROBLEM Disk Space is now: CRITICAL on wikiversity-sandbox-web.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [15:25:13] PROBLEM Free ram is now: CRITICAL on wikiversity-sandbox-web.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [15:26:43] PROBLEM Total processes is now: CRITICAL on wikiversity-sandbox-web.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [15:27:23] PROBLEM dpkg-check is now: CRITICAL on wikiversity-sandbox-web.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [15:32:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:36:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:44:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:49:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 161 processes [15:54:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 99 processes [16:02:55] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:07:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:14:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:24:33] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 17% free memory [16:29:33] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 21% free memory [16:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:37:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:44:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:48:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [17:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:07:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:15:02] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:37:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:45:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:51:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [17:56:52] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [18:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:07:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:15:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:37:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:45:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:47:33] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 13% free memory [18:47:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 155 processes [18:52:33] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 24% free memory [18:52:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 95 processes [19:07:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:07:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:17:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:37:03] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:37:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:47:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:52:33] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 8% free memory [19:55:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 216 processes [20:07:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:07:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:12:22] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 155 processes [20:17:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:37:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:38:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:47:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:07:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:08:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:17:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:37:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:38:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:47:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:07:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:08:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:17:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:32:23] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 149 processes [22:33:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 238 processes [22:36:33] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 6% free memory [22:37:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:38:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:38:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 108 processes [22:41:32] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 22% free memory [22:47:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:07:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:08:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:17:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:37:23] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:38:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:47:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26)