[00:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [00:07:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:13:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:27:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [00:34:26] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 264 MB (2% inode=84%): [00:37:16] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [00:39:46] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 21% free memory [00:43:15] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:57:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [01:07:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:14:03] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 332 processes [01:14:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:24:02] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 108 processes [01:26:52] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 10.19, 11.96, 8.26 [01:27:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [01:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 268 MB (2% inode=84%): [01:37:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [01:42:43] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [01:44:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:57:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:01:56] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 3.84, 3.67, 4.62 [02:04:26] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [02:07:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:16:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:27:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [02:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 255 MB (2% inode=84%): [02:37:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [02:46:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 18% free memory [02:46:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:48:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 170 processes [02:51:22] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 24% free memory [02:53:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 101 processes [02:57:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:04:25] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [03:07:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:16:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:29:12] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [03:30:13] PROBLEM Free ram is now: CRITICAL on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: Connection refused by host [03:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 264 MB (2% inode=84%): [03:35:13] RECOVERY Free ram is now: OK on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: OK: 93% free memory [03:37:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [03:46:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:59:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:04:32] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 292 MB (3% inode=84%): [04:07:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:16:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:29:13] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [04:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 273 MB (2% inode=84%): [04:37:43] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 21% free memory [04:37:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [04:45:43] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [04:47:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:59:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 292 MB (3% inode=84%): [05:07:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:18:14] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:29:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [05:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 271 MB (2% inode=84%): [05:37:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [05:49:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:59:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:04:26] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 292 MB (3% inode=84%): [06:07:56] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:20:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:30:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [06:31:32] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 153 processes [06:31:52] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 155 processes [06:32:12] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 154 processes [06:34:22] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 14% free memory [06:34:32] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 259 MB (2% inode=84%): [06:39:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [06:46:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [06:47:13] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 149 processes [06:50:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:51:32] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [07:00:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:04:32] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 294 MB (3% inode=84%): [07:11:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:20:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:30:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [07:34:32] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 258 MB (2% inode=84%): [07:42:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [07:50:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:00:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 294 MB (3% inode=84%): [08:12:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:20:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:30:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [08:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 270 MB (2% inode=84%): [08:40:46] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 21% free memory [08:42:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [08:50:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:59:23] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 24% free memory [09:01:32] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:04:32] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [09:08:53] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [09:12:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:21:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:32:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [09:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 270 MB (2% inode=84%): [09:42:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [09:51:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:02:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [10:12:56] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:21:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:32:25] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [10:34:25] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 258 MB (2% inode=84%): [10:47:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [10:51:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:02:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [11:07:23] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 16% free memory [11:12:23] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 23% free memory [11:17:12] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:21:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:32:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [11:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 265 MB (2% inode=84%): [11:47:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [11:53:15] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:00:43] PROBLEM dpkg-check is now: CRITICAL on wikistats-history-01.pmtpa.wmflabs 10.4.0.111 output: DPKG CRITICAL dpkg reports broken packages [12:02:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 293 MB (3% inode=84%): [12:17:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:24:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [12:25:56] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:26:26] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 14% free memory [12:27:56] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 181 processes [12:32:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [12:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 255 MB (2% inode=84%): [12:37:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 102 processes [12:38:53] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 21% free memory [12:47:42] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [12:56:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:01:53] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [13:02:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 292 MB (3% inode=84%): [13:17:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:26:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:32:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [13:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 279 MB (2% inode=84%): [13:34:33] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [13:47:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [13:56:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:02:53] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 292 MB (3% inode=84%): [14:17:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:26:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:34:22] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [14:34:32] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 267 MB (2% inode=84%): [14:47:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [14:58:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:04:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 292 MB (3% inode=84%): [15:17:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:28:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:34:25] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [15:34:25] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 276 MB (2% inode=84%): [15:45:53] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 152 processes [15:47:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [15:55:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 150 processes [15:58:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:04:32] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 292 MB (3% inode=84%): [16:04:42] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:17:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:23:52] PROBLEM Current Load is now: CRITICAL on mwreview-dev1.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:24:32] PROBLEM Disk Space is now: CRITICAL on mwreview-dev1.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:25:12] PROBLEM Free ram is now: CRITICAL on mwreview-dev1.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:26:42] PROBLEM Total processes is now: CRITICAL on mwreview-dev1.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:27:22] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev1.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [16:28:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 258 MB (2% inode=84%): [16:34:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [16:39:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.1.113 output: Connection refused by host [16:40:13] PROBLEM Free ram is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.1.113 output: Connection refused by host [16:40:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.1.113 output: Connection refused by host [16:41:43] PROBLEM Total processes is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.1.113 output: Connection refused by host [16:41:53] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 21% free memory [16:42:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev2.pmtpa.wmflabs 10.4.1.113 output: Connection refused by host [16:47:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [16:48:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: Connection refused by host [16:49:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: Connection refused by host [16:50:13] PROBLEM Free ram is now: CRITICAL on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: Connection refused by host [16:51:43] PROBLEM Total processes is now: CRITICAL on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: Connection refused by host [16:52:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev3.pmtpa.wmflabs 10.4.1.117 output: Connection refused by host [16:54:53] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [16:58:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 291 MB (3% inode=84%): [17:04:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:21:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:26:53] RECOVERY Total processes is now: OK on deployment-el-test.pmtpa.wmflabs 10.4.1.105 output: PROCS OK: 83 processes [17:28:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:28:53] RECOVERY Current Load is now: OK on deployment-el-test.pmtpa.wmflabs 10.4.1.105 output: OK - load average: 0.11, 0.19, 0.09 [17:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 255 MB (2% inode=84%): [17:34:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [17:38:53] PROBLEM Current Load is now: CRITICAL on deployment-el-test2.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [17:39:33] PROBLEM Disk Space is now: CRITICAL on deployment-el-test2.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [17:40:13] PROBLEM Free ram is now: CRITICAL on deployment-el-test2.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [17:41:43] PROBLEM Total processes is now: CRITICAL on deployment-el-test2.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [17:42:23] PROBLEM dpkg-check is now: CRITICAL on deployment-el-test2.pmtpa.wmflabs 10.4.1.108 output: Connection refused by host [17:43:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 250 processes [17:47:32] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 6% free memory [17:52:03] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [17:53:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 102 processes [17:54:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [17:55:13] PROBLEM Free ram is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [17:55:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [17:56:43] PROBLEM Total processes is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [17:57:23] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev4.pmtpa.wmflabs 10.4.1.105 output: Connection refused by host [17:57:33] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 23% free memory [17:59:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 291 MB (3% inode=84%): [18:04:43] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [18:30:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:34:32] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 264 MB (2% inode=84%): [18:35:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [18:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:01:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:04:32] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 291 MB (3% inode=84%): [19:05:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:22:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [19:31:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:34:32] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 274 MB (2% inode=84%): [19:35:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [19:52:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:01:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 291 MB (3% inode=84%): [20:04:33] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 151 processes [20:05:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:14:33] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [20:22:22] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:31:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 261 MB (2% inode=84%): [20:35:23] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [20:39:53] RECOVERY Free ram is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: OK: 21% free memory [20:52:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [20:57:53] PROBLEM Free ram is now: WARNING on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: Warning: 19% free memory [21:01:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 290 MB (3% inode=84%): [21:05:24] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:22:33] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [21:31:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 267 MB (2% inode=84%): [21:36:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [21:43:59] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 155 processes [21:52:52] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:01:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 290 MB (3% inode=84%): [22:07:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:13:53] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 148 processes [22:22:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [22:31:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 259 MB (2% inode=84%): [22:37:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [22:52:53] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:01:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:04:23] PROBLEM Disk Space is now: WARNING on flog.pmtpa.wmflabs 10.4.1.104 output: DISK WARNING - free space: / 290 MB (3% inode=84%): [23:07:33] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:21:52] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 157 processes [23:27:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253) [23:31:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:34:23] PROBLEM Disk Space is now: CRITICAL on flog.pmtpa.wmflabs 10.4.1.104 output: DISK CRITICAL - free space: / 272 MB (2% inode=84%): [23:37:52] PROBLEM host: parsoid-roundtrip7-8core.pmtpa.wmflabs is DOWN address: 10.4.1.26 CRITICAL - Host Unreachable (10.4.1.26) [23:57:13] PROBLEM host: fundraising-dev-awight.pmtpa.wmflabs is DOWN address: 10.4.0.253 CRITICAL - Host Unreachable (10.4.0.253)