[00:21:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:54:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:24:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:54:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:24:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:54:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:26:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:30:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 246 processes [03:40:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 153 processes [03:45:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 95 processes [03:56:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:28:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:59:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:29:44] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:01:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:23:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 237 processes [06:31:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:33:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 171 processes [06:38:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 140 processes [07:01:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:34:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:04:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:13:52] PROBLEM Current Load is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [08:14:32] PROBLEM Disk Space is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [08:15:12] PROBLEM Free ram is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [08:16:42] PROBLEM Total processes is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [08:17:22] PROBLEM dpkg-check is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [08:34:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:53:43] RECOVERY Total processes is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS OK: 97 processes [08:53:43] RECOVERY Free ram is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK: 98% free memory [08:54:13] RECOVERY Current Load is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: OK - load average: 0.15, 0.11, 0.05 [08:56:32] RECOVERY Total processes is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: PROCS OK: 96 processes [08:56:32] RECOVERY dpkg-check is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: All packages OK [08:56:52] RECOVERY Disk Space is now: OK on dumps-bot3.pmtpa.wmflabs 10.4.0.118 output: DISK OK [09:00:53] RECOVERY Total processes is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS OK: 98 processes [09:00:53] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [09:00:53] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 97% free memory [09:01:53] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [09:01:53] RECOVERY SSH is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [09:02:53] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 0.06, 0.31, 0.18 [09:04:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:07:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 152 processes [09:17:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 104 processes [09:36:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:39:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:09:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:39:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:09:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:39:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:03:12] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 10% free memory [13:08:12] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 1% free memory [13:11:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:13:12] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 78% free memory [13:41:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:13:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:44:12] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:06:44] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 150 processes [15:14:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:35:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 196 processes [15:44:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:45:52] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 105 processes [15:56:42] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 153 processes [16:06:43] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 150 processes [16:10:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 168 processes [16:16:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:46:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:06:33] RECOVERY Disk Space is now: OK on moodle-24.pmtpa.wmflabs 10.4.0.92 output: DISK OK [17:06:43] RECOVERY Total processes is now: OK on moodle-24.pmtpa.wmflabs 10.4.0.92 output: PROCS OK: 93 processes [17:07:23] PROBLEM Free ram is now: UNKNOWN on moodle-24.pmtpa.wmflabs 10.4.0.92 output: NRPE: Unable to read output [17:07:33] RECOVERY dpkg-check is now: OK on moodle-24.pmtpa.wmflabs 10.4.0.92 output: All packages OK [17:10:53] RECOVERY Current Load is now: OK on moodle-24.pmtpa.wmflabs 10.4.0.92 output: OK - load average: 0.05, 0.19, 0.11 [17:19:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:49:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:50:43] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 24% free memory [18:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:08:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 193 processes [19:18:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 106 processes [19:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:33:52] PROBLEM Current Load is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:34:32] PROBLEM Disk Space is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:35:13] PROBLEM Free ram is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:36:43] PROBLEM Total processes is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:37:23] PROBLEM dpkg-check is now: CRITICAL on openstack-role-dev7.pmtpa.wmflabs 10.4.1.108 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:39:33] PROBLEM Disk Space is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [19:40:23] PROBLEM Free ram is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [19:40:53] PROBLEM Current Load is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [19:41:43] PROBLEM Total processes is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [19:42:23] PROBLEM dpkg-check is now: CRITICAL on math-test.pmtpa.wmflabs 10.4.1.107 output: Connection refused by host [19:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:54:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 7% free memory [20:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:01:52] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 202 processes [21:11:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 104 processes [21:19:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:49:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:13:53] PROBLEM Current Load is now: CRITICAL on he-moodle-24.pmtpa.wmflabs 10.4.1.109 output: Connection refused by host [22:14:33] PROBLEM Disk Space is now: CRITICAL on he-moodle-24.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:15:12] PROBLEM Free ram is now: CRITICAL on he-moodle-24.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:16:42] PROBLEM Total processes is now: CRITICAL on he-moodle-24.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:17:22] PROBLEM dpkg-check is now: CRITICAL on he-moodle-24.pmtpa.wmflabs 10.4.1.109 output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:21:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:51:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:21:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:52:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3)