[00:01:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:31:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:02:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:13:53] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 306 processes [01:18:53] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 102 processes [01:33:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:04:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:34:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:04:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:07:23] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 48% free memory [03:34:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:04:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:28:52] PROBLEM Current Load is now: CRITICAL on en.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [04:29:32] PROBLEM Disk Space is now: CRITICAL on en.pmtpa.wmflabs 10.4.0.100 output: Connection refused by host [04:30:13] PROBLEM Free ram is now: CRITICAL on en.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:30:13] PROBLEM Disk Space is now: WARNING on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: DISK WARNING - free space: / 555 MB (5% inode=80%): [04:31:43] PROBLEM Total processes is now: CRITICAL on en.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:32:23] PROBLEM dpkg-check is now: CRITICAL on en.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:34:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:53:53] PROBLEM Current Load is now: CRITICAL on voy-en.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [04:54:33] PROBLEM Disk Space is now: CRITICAL on voy-en.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [04:55:13] PROBLEM Free ram is now: CRITICAL on voy-en.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [04:56:43] PROBLEM Total processes is now: CRITICAL on voy-en.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [04:57:23] PROBLEM dpkg-check is now: CRITICAL on voy-en.pmtpa.wmflabs 10.4.0.251 output: Connection refused by host [05:05:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:06:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 198 processes [06:11:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [06:21:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 198 processes [06:34:32] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [06:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:40:13] PROBLEM Disk Space is now: CRITICAL on maps-tiles1.pmtpa.wmflabs 10.4.0.136 output: DISK CRITICAL - free space: / 267 MB (2% inode=80%): [06:40:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 199 processes [06:58:53] PROBLEM Current Load is now: CRITICAL on latexml-test.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:59:33] PROBLEM Disk Space is now: CRITICAL on latexml-test.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:00:22] PROBLEM Free ram is now: CRITICAL on latexml-test.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:01:42] PROBLEM Total processes is now: CRITICAL on latexml-test.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:02:22] PROBLEM dpkg-check is now: CRITICAL on latexml-test.pmtpa.wmflabs 10.4.0.100 output: CHECK_NRPE: Error - Could not complete SSL handshake. [07:05:33] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [07:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:24:33] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 149 processes [07:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:36:33] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [08:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:07:13] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [08:37:52] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [08:37:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:08:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [09:08:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:38:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [09:39:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:08:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [10:09:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:38:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [10:39:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:08:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [11:09:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:38:24] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [11:39:44] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:09:03] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [12:10:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:39:03] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [12:41:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:09:12] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [13:11:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:39:13] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [13:41:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:46:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 252 processes [13:51:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 109 processes [14:09:32] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [14:11:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:40:42] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [14:44:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:11:43] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [15:14:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:42:03] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [15:44:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:12:03] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [16:14:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:42:13] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [16:44:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:12:52] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [17:15:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:42:52] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [17:46:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:13:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [18:17:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:43:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [18:48:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:13:23] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [19:19:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:24:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 176 processes [19:29:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 100 processes [19:44:02] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [19:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:01:43] RECOVERY Total processes is now: OK on voy-en.pmtpa.wmflabs 10.4.0.251 output: PROCS OK: 92 processes [20:02:33] RECOVERY dpkg-check is now: OK on voy-en.pmtpa.wmflabs 10.4.0.251 output: All packages OK [20:03:53] RECOVERY Current Load is now: OK on voy-en.pmtpa.wmflabs 10.4.0.251 output: OK - load average: 0.16, 0.22, 0.10 [20:04:33] RECOVERY Disk Space is now: OK on voy-en.pmtpa.wmflabs 10.4.0.251 output: DISK OK [20:05:13] PROBLEM Free ram is now: UNKNOWN on voy-en.pmtpa.wmflabs 10.4.0.251 output: NRPE: Unable to read output [20:14:03] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [20:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:44:03] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [20:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:14:12] PROBLEM host: etherpad-lite.pmtpa.wmflabs is DOWN address: 10.4.0.87 CRITICAL - Host Unreachable (10.4.0.87) [21:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:29:12] RECOVERY host: etherpad-lite.pmtpa.wmflabs is UP address: 10.4.0.87 PING OK - Packet loss = 0%, RTA = 2.17 ms [21:30:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 355 processes [21:31:43] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 357 processes [21:37:33] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 317 processes [21:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:50:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 172 processes [21:51:43] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 165 processes [21:52:33] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 146 processes [22:08:53] PROBLEM Current Load is now: CRITICAL on maps-warper.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [22:09:33] PROBLEM Disk Space is now: CRITICAL on maps-warper.pmtpa.wmflabs 10.4.1.99 output: Connection refused by host [22:10:12] PROBLEM Free ram is now: CRITICAL on maps-warper.pmtpa.wmflabs 10.4.1.99 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:11:42] PROBLEM Total processes is now: CRITICAL on maps-warper.pmtpa.wmflabs 10.4.1.99 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:12:22] PROBLEM dpkg-check is now: CRITICAL on maps-warper.pmtpa.wmflabs 10.4.1.99 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:07:57] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 202 processes [23:12:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 187 processes [23:17:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 101 processes [23:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3)