[00:02:52] RECOVERY SSH is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:10:52] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [00:12:52] PROBLEM dpkg-check is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [00:17:54] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:23:44] PROBLEM Current Load is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [00:33:46] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:10:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 340 processes [01:14:03] PROBLEM Free ram is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Warning: 13% free memory [01:15:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 109 processes [01:19:03] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: Critical: 5% free memory [01:24:12] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 85% free memory [02:07:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [02:15:44] PROBLEM Current Load is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [02:29:44] PROBLEM Total processes is now: UNKNOWN on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: NRPE: Call to fork() failed [02:32:52] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:34:42] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:36:44] PROBLEM Disk Space is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:37:34] PROBLEM Current Load is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [02:37:44] PROBLEM SSH is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Server answer: [02:41:43] RECOVERY Disk Space is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DISK OK [02:42:03] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 26% free memory [02:42:33] RECOVERY Current Load is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK - load average: 2.39, 2.16, 1.53 [02:42:43] RECOVERY SSH is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [02:42:53] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [02:44:43] PROBLEM Total processes is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS WARNING: 190 processes [02:47:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [02:52:54] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [03:02:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [03:02:52] RECOVERY SSH is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [03:12:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [03:19:23] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:20:54] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [03:44:22] PROBLEM Free ram is now: UNKNOWN on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: NRPE: Call to fork() failed [03:54:22] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:18:54] PROBLEM SSH is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Server answer: [05:27:14] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [05:27:34] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [05:32:14] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [05:32:34] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [05:39:25] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:56:54] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:57:14] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:58:34] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [05:59:44] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: CHECK_NRPE: Error - Could not complete SSL handshake. [06:17:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 211 processes [06:27:46] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 198 processes [08:47:14] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [08:52:14] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [09:36:14] PROBLEM Free ram is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: Warning: 19% free memory [10:42:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 190 processes [11:01:23] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: Critical: 5% free memory [11:06:12] PROBLEM Free ram is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: Warning: 14% free memory [11:07:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 208 processes [11:21:14] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: Critical: 5% free memory [11:44:43] PROBLEM SSH is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: CRITICAL - Socket timeout after 10 seconds [11:47:04] PROBLEM Current Load is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: CHECK_NRPE: Socket timeout after 10 seconds. [11:47:24] PROBLEM Disk Space is now: UNKNOWN on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: NRPE: Call to fork() failed [11:47:34] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: CHECK_NRPE: Socket timeout after 10 seconds. [11:49:34] RECOVERY SSH is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [11:51:52] RECOVERY Total processes is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS OK: 110 processes [11:52:22] RECOVERY Disk Space is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: DISK OK [11:52:22] RECOVERY dpkg-check is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: All packages OK [11:56:53] PROBLEM Current Load is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: WARNING - load average: 0.03, 12.38, 18.27 [12:17:14] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:18:34] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:21:54] RECOVERY Current Load is now: OK on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: OK - load average: 1.48, 0.69, 4.00 [12:22:14] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [12:23:34] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [13:44:43] PROBLEM Total processes is now: WARNING on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS WARNING: 153 processes [14:27:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 247 processes [14:57:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [15:17:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [15:44:52] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9.pmtpa.wmflabs 10.4.1.41 output: PROCS CRITICAL: 202 processes [15:47:14] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [15:47:44] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 199 processes [15:52:12] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [15:57:14] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [17:37:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [17:42:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 168 processes [17:47:14] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 126 processes [18:02:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 193 processes [19:08:14] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 152 processes [19:32:12] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [19:34:32] PROBLEM dpkg-check is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Unable to read output [19:37:15] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [19:39:35] RECOVERY dpkg-check is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: All packages OK [20:13:13] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 189 processes [21:46:14] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 154 processes [23:02:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 205 processes [23:22:14] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [23:27:13] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory