[00:22:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [00:28:32] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [00:41:22] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: Critical: 5% free memory [00:50:29] PROBLEM Free ram is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: Warning: 12% free memory [01:07:47] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: Critical: 3% free memory [01:19:48] RECOVERY dpkg-check is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: All packages OK [01:19:48] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 626 processes [01:19:48] RECOVERY SSH is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [01:19:48] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 139 processes [01:19:48] PROBLEM Free ram is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: Critical: 5% free memory [01:19:48] RECOVERY Disk Space is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: DISK OK [01:19:48] PROBLEM Current Load is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: WARNING - load average: 0.08, 12.85, 18.12 [01:19:48] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 114 processes [01:19:48] RECOVERY Free ram is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: OK: 91% free memory [01:22:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [01:26:42] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [01:41:07] RECOVERY Current Load is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: OK - load average: 1.07, 0.40, 3.81 [02:53:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 159 processes [03:11:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 177 processes [03:37:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [03:40:53] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS CRITICAL: 205 processes [03:45:52] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 199 processes [03:50:53] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS CRITICAL: 203 processes [04:12:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [04:44:23] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 179 processes [04:54:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 114 processes [06:02:12] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [06:12:12] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [06:22:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [06:49:33] RECOVERY Total processes is now: OK on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS OK: 150 processes [07:12:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [07:22:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [08:10:52] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 94 processes [08:13:32] RECOVERY Disk Space is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: DISK OK [08:13:32] RECOVERY Total processes is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: PROCS OK: 107 processes [08:13:52] RECOVERY Current Load is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: OK - load average: 0.19, 0.12, 0.05 [08:16:53] RECOVERY Free ram is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: OK: 94% free memory [08:16:53] RECOVERY dpkg-check is now: OK on wikidata-testrepo.pmtpa.wmflabs 10.4.0.164 output: All packages OK [08:55:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [09:12:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [09:17:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 193 processes [10:07:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [10:15:52] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 152 processes [10:42:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 184 processes [11:33:52] RECOVERY Current Load is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: OK - load average: 0.41, 0.21, 0.08 [11:33:52] RECOVERY Total processes is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: PROCS OK: 96 processes [11:34:52] RECOVERY dpkg-check is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: All packages OK [11:35:13] RECOVERY Free ram is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: OK: 86% free memory [11:36:53] RECOVERY Disk Space is now: OK on wikidata-client-test.pmtpa.wmflabs 10.4.1.3 output: DISK OK [12:00:32] RECOVERY Total processes is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: PROCS OK: 81 processes [12:00:52] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS CRITICAL: 203 processes [12:01:12] RECOVERY Total processes is now: OK on patchtest.pmtpa.wmflabs 10.4.0.69 output: PROCS OK: 82 processes [12:01:42] RECOVERY Disk Space is now: OK on patchtest.pmtpa.wmflabs 10.4.0.69 output: DISK OK [12:01:42] RECOVERY Current Load is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: OK - load average: 0.00, 0.00, 0.00 [12:01:42] RECOVERY Free ram is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: OK: 89% free memory [12:05:53] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 196 processes [12:08:33] PROBLEM Total processes is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:23] PROBLEM Total processes is now: CRITICAL on patchtest.pmtpa.wmflabs 10.4.0.69 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:53] PROBLEM Disk Space is now: CRITICAL on patchtest.pmtpa.wmflabs 10.4.0.69 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:53] PROBLEM Current Load is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:53] PROBLEM Free ram is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:52] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS CRITICAL: 210 processes [12:33:22] RECOVERY Total processes is now: OK on patchtest2.pmtpa.wmflabs 10.4.0.74 output: PROCS OK: 81 processes [13:14:43] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 184 processes [13:17:42] RECOVERY Total processes is now: OK on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS OK: 148 processes [13:21:33] PROBLEM Total processes is now: CRITICAL on patchtest2.pmtpa.wmflabs 10.4.0.74 output: CHECK_NRPE: Socket timeout after 10 seconds. [13:29:42] PROBLEM Total processes is now: CRITICAL on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS CRITICAL: 220 processes [13:30:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 184 processes [13:40:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 214 processes [16:13:53] PROBLEM Current Load is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [16:14:33] PROBLEM Disk Space is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [16:15:12] PROBLEM Free ram is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [16:16:42] PROBLEM Total processes is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [16:17:22] PROBLEM dpkg-check is now: CRITICAL on mwreview-dev8.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [17:34:42] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [17:35:54] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 33% free memory [17:39:44] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 192 processes [20:22:33] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [20:32:12] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 93% free memory [20:33:22] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 89% free memory [20:34:22] RECOVERY Free ram is now: OK on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: OK: 92% free memory [20:37:33] PROBLEM Total processes is now: CRITICAL on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS CRITICAL: 276 processes [20:58:53] PROBLEM Current Load is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: Connection refused by host [20:58:53] PROBLEM Disk Space is now: CRITICAL on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: Connection refused by host [20:58:54] PROBLEM Free ram is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [20:59:33] PROBLEM Disk Space is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: Connection refused by host [20:59:33] PROBLEM Free ram is now: CRITICAL on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: Connection refused by host [21:00:12] PROBLEM Free ram is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: Connection refused by host [21:00:22] PROBLEM Total processes is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:00:52] PROBLEM dpkg-check is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:01:02] PROBLEM Total processes is now: CRITICAL on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: Connection refused by host [21:01:32] PROBLEM Total processes is now: CRITICAL on solr2.pmtpa.wmflabs 10.4.0.250 output: Connection refused by host [21:01:42] PROBLEM Total processes is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: Connection refused by host [21:01:42] PROBLEM dpkg-check is now: CRITICAL on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: Connection refused by host [21:01:42] PROBLEM Current Load is now: CRITICAL on solr2.pmtpa.wmflabs 10.4.0.250 output: Connection refused by host [21:02:22] PROBLEM dpkg-check is now: CRITICAL on solr2.pmtpa.wmflabs 10.4.0.250 output: Connection refused by host [21:02:22] PROBLEM dpkg-check is now: CRITICAL on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: Connection refused by host [21:02:22] PROBLEM Current Load is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:02:22] PROBLEM Disk Space is now: CRITICAL on solr2.pmtpa.wmflabs 10.4.0.250 output: Connection refused by host [21:03:12] PROBLEM Current Load is now: CRITICAL on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: Connection refused by host [21:03:12] PROBLEM Disk Space is now: CRITICAL on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: Connection refused by host [21:03:12] PROBLEM Free ram is now: CRITICAL on solr2.pmtpa.wmflabs 10.4.0.250 output: Connection refused by host [21:03:52] PROBLEM Disk Space is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:04:32] PROBLEM Disk Space is now: CRITICAL on solr-mw.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:04:32] PROBLEM Disk Space is now: CRITICAL on solr1.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [21:04:32] PROBLEM Free ram is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:05:13] PROBLEM Free ram is now: CRITICAL on solr-mw.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:05:13] PROBLEM Free ram is now: CRITICAL on solr1.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [21:05:53] PROBLEM Current Load is now: CRITICAL on solr-mw.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:05:53] PROBLEM Current Load is now: CRITICAL on solr1.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [21:06:03] PROBLEM Free ram is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.21 output: Connection refused by host [21:06:03] PROBLEM Total processes is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.42 output: Connection refused by host [21:06:33] PROBLEM Current Load is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.21 output: Connection refused by host [21:06:43] PROBLEM Total processes is now: CRITICAL on solr-mw.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:06:44] PROBLEM Total processes is now: CRITICAL on solr1.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [21:06:44] PROBLEM dpkg-check is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.42 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:07:13] PROBLEM Disk Space is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.21 output: Connection refused by host [21:07:23] PROBLEM dpkg-check is now: CRITICAL on solr-mw.pmtpa.wmflabs 10.4.1.46 output: Connection refused by host [21:07:23] PROBLEM Total processes is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.21 output: Connection refused by host [21:07:23] PROBLEM dpkg-check is now: CRITICAL on solr1.pmtpa.wmflabs 10.4.0.246 output: Connection refused by host [21:07:53] PROBLEM dpkg-check is now: CRITICAL on solr0.pmtpa.wmflabs 10.4.1.21 output: Connection refused by host [21:08:03] PROBLEM Current Load is now: CRITICAL on solr3.pmtpa.wmflabs 10.4.1.42 output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:32:23] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [22:04:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.27, 1.61, 4.84 [22:07:33] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [22:09:33] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 4.10, 3.07, 4.87 [22:17:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 7.09, 5.90, 5.48 [22:22:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 9.17, 7.92, 6.10 [22:31:43] RECOVERY dpkg-check is now: OK on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: All packages OK [22:31:43] RECOVERY Total processes is now: OK on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: PROCS OK: 96 processes [22:32:23] RECOVERY dpkg-check is now: OK on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: All packages OK [22:32:23] RECOVERY Current Load is now: OK on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: OK - load average: 1.69, 0.48, 0.16 [22:33:13] RECOVERY Current Load is now: OK on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: OK - load average: 1.06, 0.67, 0.27 [22:33:13] RECOVERY Disk Space is now: OK on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: DISK OK [22:33:53] RECOVERY Current Load is now: OK on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: OK - load average: 0.26, 0.36, 0.16 [22:33:53] RECOVERY Disk Space is now: OK on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: DISK OK [22:33:53] PROBLEM Free ram is now: UNKNOWN on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: NRPE: Unable to read output [22:34:33] RECOVERY Disk Space is now: OK on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: DISK OK [22:34:33] PROBLEM Free ram is now: UNKNOWN on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: NRPE: Unable to read output [22:34:43] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK CRITICAL - free space: /tmp 0 MB (0% inode=99%): [22:35:12] PROBLEM Free ram is now: UNKNOWN on solr-zk0.pmtpa.wmflabs 10.4.0.216 output: NRPE: Unable to read output [22:35:22] RECOVERY Total processes is now: OK on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: PROCS OK: 87 processes [22:35:52] RECOVERY dpkg-check is now: OK on solr-zk2.pmtpa.wmflabs 10.4.0.203 output: All packages OK [22:36:02] RECOVERY Total processes is now: OK on solr-zk1.pmtpa.wmflabs 10.4.0.70 output: PROCS OK: 85 processes [22:51:22] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 208 processes [22:53:12] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 18% free memory [22:56:23] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 103 processes [23:03:13] RECOVERY Free ram is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK: 20% free memory