[00:23:43] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:54:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [00:56:12] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 27% free memory [01:24:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [01:54:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:24:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [02:54:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:25:02] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [03:56:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:26:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [04:56:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:26:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [05:29:23] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 23% free memory [05:59:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:31:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [06:32:32] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 151 processes [06:42:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 16% free memory [06:52:33] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 146 processes [06:57:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 20% free memory [07:01:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [07:11:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 19% free memory [07:31:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:04:42] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [08:34:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:04:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [09:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [10:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [11:10:53] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 332 processes [11:15:32] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 305 processes [11:16:42] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 318 processes [11:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [12:12:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 157 processes [12:17:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 118 processes [12:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [13:36:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:06:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [14:38:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:09:33] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [15:39:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:09:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:39:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [16:53:23] PROBLEM Free ram is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: Warning: 19% free memory [17:09:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [17:39:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:09:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:41:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [18:53:53] PROBLEM Current Load is now: CRITICAL on gerrit-test.pmtpa.wmflabs 10.4.1.110 output: Connection refused by host [18:54:33] PROBLEM Disk Space is now: CRITICAL on gerrit-test.pmtpa.wmflabs 10.4.1.110 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:55:13] PROBLEM Free ram is now: CRITICAL on gerrit-test.pmtpa.wmflabs 10.4.1.110 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:56:43] PROBLEM Total processes is now: CRITICAL on gerrit-test.pmtpa.wmflabs 10.4.1.110 output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:57:23] PROBLEM dpkg-check is now: CRITICAL on gerrit-test.pmtpa.wmflabs 10.4.1.110 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:11:22] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [19:11:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory [19:28:53] PROBLEM Current Load is now: CRITICAL on deployment-solr.pmtpa.wmflabs 10.4.1.111 output: Connection refused by host [19:29:33] PROBLEM Disk Space is now: CRITICAL on deployment-solr.pmtpa.wmflabs 10.4.1.111 output: Connection refused by host [19:30:12] PROBLEM Free ram is now: CRITICAL on deployment-solr.pmtpa.wmflabs 10.4.1.111 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:31:42] PROBLEM Total processes is now: CRITICAL on deployment-solr.pmtpa.wmflabs 10.4.1.111 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:32:22] PROBLEM dpkg-check is now: CRITICAL on deployment-solr.pmtpa.wmflabs 10.4.1.111 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:41:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:14:52] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:25:53] PROBLEM Current Load is now: CRITICAL on kirke.pmtpa.wmflabs 10.4.1.112 output: Connection refused by host [20:26:33] PROBLEM Disk Space is now: CRITICAL on kirke.pmtpa.wmflabs 10.4.1.112 output: Connection refused by host [20:26:43] PROBLEM Total processes is now: CRITICAL on kirke.pmtpa.wmflabs 10.4.1.112 output: Connection refused by host [20:27:13] PROBLEM Free ram is now: CRITICAL on kirke.pmtpa.wmflabs 10.4.1.112 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:27:23] PROBLEM dpkg-check is now: CRITICAL on kirke.pmtpa.wmflabs 10.4.1.112 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:37:53] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 244 processes [20:42:13] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 17% free memory [20:44:53] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [20:50:33] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 200 processes [20:52:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 25% free memory [20:52:53] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 164 processes [21:07:58] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 97 processes [21:16:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [21:46:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:16:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [22:46:23] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:16:32] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3) [23:47:13] PROBLEM host: wikidata-client-test.pmtpa.wmflabs is DOWN address: 10.4.1.3 CRITICAL - Host Unreachable (10.4.1.3)