[00:05:09] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:09:49] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [00:26:29] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [00:29:49] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 26% free memory [00:49:31] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [00:53:49] PROBLEM Total Processes is now: CRITICAL on incubator-bot2 i-00000252 output: PROCS CRITICAL: 201 processes [00:56:29] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [01:09:39] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [01:14:21] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [01:26:46] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [01:33:00] PROBLEM dpkg-check is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:37:49] RECOVERY dpkg-check is now: OK on deployment-apache30 i-000002d3 output: All packages OK [01:45:03] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [01:49:53] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [01:56:48] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:26:52] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:40:35] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 200 processes [02:53:52] 07/04/2012 - 02:53:52 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [02:55:44] RECOVERY Puppet freshness is now: OK on bots-sql2 i-000000af output: puppet ran at Wed Jul 4 02:55:30 UTC 2012 [02:56:52] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:57:06] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [03:01:53] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [03:25:14] PROBLEM Free ram is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:27:13] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [03:30:03] PROBLEM Free ram is now: UNKNOWN on psm-precise i-000002f2 output: NRPE: Unable to read output [03:43:58] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: CHECK_NRPE: Socket timeout after 10 seconds. [03:45:51] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [03:47:34] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:48:51] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:41] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 6% free memory [03:50:21] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [03:51:40] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [03:52:10] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 2.69, 5.01, 3.83 [03:53:30] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 17% free memory [03:53:31] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 13% free memory [03:54:10] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 41% free memory [03:57:10] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.65, 2.25, 2.95 [03:58:11] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [04:03:20] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [04:13:30] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 4% free memory [04:13:30] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:18:20] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:18:31] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:23:30] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:27:11] PROBLEM Total Processes is now: WARNING on incubator-bot1 i-00000251 output: PROCS WARNING: 151 processes [04:28:11] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [04:28:26] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:59:38] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [05:23:18] PROBLEM Puppet freshness is now: CRITICAL on deployment-transcoding i-00000105 output: Puppet has not run in last 20 hours [05:25:18] PROBLEM Puppet freshness is now: CRITICAL on gerrit i-000000ff output: Puppet has not run in last 20 hours [05:28:08] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [05:29:46] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [05:31:56] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 201 processes [05:32:21] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [05:37:01] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 200 processes [05:37:10] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 7% free memory [05:41:44] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [05:46:42] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [05:52:25] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [05:57:06] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [06:00:16] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [06:02:36] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [06:07:10] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 206 processes [06:33:09] PROBLEM Current Users is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:09] PROBLEM Current Load is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:09] PROBLEM Free ram is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:09] PROBLEM Disk Space is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:34:27] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 8.87, 8.75, 4.94 [06:34:32] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [06:34:54] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:35:04] PROBLEM dpkg-check is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:35:04] PROBLEM Total Processes is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:37] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:36:43] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [06:42:06] RECOVERY Current Users is now: OK on deployment-apache31 i-000002d4 output: USERS OK - 0 users currently logged in [06:42:06] RECOVERY Current Load is now: OK on deployment-apache31 i-000002d4 output: OK - load average: 2.27, 3.05, 2.48 [06:42:06] RECOVERY Disk Space is now: OK on deployment-apache31 i-000002d4 output: DISK OK [06:42:06] RECOVERY Free ram is now: OK on deployment-apache31 i-000002d4 output: OK: 93% free memory [06:43:38] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:38] PROBLEM Free ram is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:18] RECOVERY Total Processes is now: OK on deployment-apache31 i-000002d4 output: PROCS OK: 128 processes [06:44:25] RECOVERY dpkg-check is now: OK on deployment-apache31 i-000002d4 output: All packages OK [06:45:20] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:05] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [06:48:20] PROBLEM Current Load is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:09] PROBLEM Free ram is now: CRITICAL on aggregator-test1 i-000002bf output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:01] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:27] PROBLEM Free ram is now: UNKNOWN on psm-precise i-000002f2 output: NRPE: Unable to read output [06:56:58] PROBLEM Free ram is now: WARNING on aggregator-test1 i-000002bf output: Warning: 6% free memory [06:56:58] RECOVERY Current Load is now: OK on upload-wizard i-0000021c output: OK - load average: 0.05, 0.98, 2.24 [06:56:58] RECOVERY Disk Space is now: OK on upload-wizard i-0000021c output: DISK OK [06:57:08] PROBLEM SSH is now: CRITICAL on bots-sql2 i-000000af output: CRITICAL - Socket timeout after 10 seconds [06:57:08] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:58:15] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:20] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 2.63, 4.11, 5.56 [07:00:04] PROBLEM Current Load is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:12] PROBLEM Free ram is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:33] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:38] PROBLEM Free ram is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:07] PROBLEM Current Load is now: WARNING on redis1 i-000002b6 output: WARNING - load average: 2.08, 4.52, 5.08 [07:05:07] PROBLEM Current Load is now: WARNING on wikidata-dev-2 i-00000259 output: WARNING - load average: 3.00, 6.43, 7.13 [07:05:14] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [07:05:14] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [07:05:33] PROBLEM Total Processes is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:14] PROBLEM Disk Space is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:14] PROBLEM Current Users is now: CRITICAL on fr-wiki-db-precise i-0000023e output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:14] PROBLEM Disk Space is now: CRITICAL on fr-wiki-db-precise i-0000023e output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:14] PROBLEM Total Processes is now: CRITICAL on fr-wiki-db-precise i-0000023e output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:24] PROBLEM dpkg-check is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:07:25] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [07:07:59] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [07:08:04] RECOVERY Current Load is now: OK on pybal-precise i-00000289 output: OK - load average: 0.86, 3.06, 3.42 [07:08:04] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [07:10:00] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 6.27, 3.79, 4.34 [07:10:00] RECOVERY Free ram is now: OK on pybal-precise i-00000289 output: OK: 77% free memory [07:10:16] PROBLEM Current Load is now: WARNING on labs-nfs1 i-0000005d output: WARNING - load average: 5.56, 5.98, 5.73 [07:10:44] PROBLEM Total Processes is now: WARNING on incubator-bot1 i-00000251 output: PROCS WARNING: 158 processes [07:10:51] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 10.94, 13.56, 15.07 [07:11:30] PROBLEM Disk Space is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM Current Users is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM Free ram is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:30] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:49] PROBLEM Disk Space is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:49] PROBLEM Current Users is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:49] PROBLEM Current Load is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:50] PROBLEM Total Processes is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:55] PROBLEM dpkg-check is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:55] PROBLEM Free ram is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:57] RECOVERY Current Load is now: OK on redis1 i-000002b6 output: OK - load average: 0.04, 0.87, 2.94 [07:12:57] RECOVERY Current Users is now: OK on fr-wiki-db-precise i-0000023e output: USERS OK - 0 users currently logged in [07:12:57] RECOVERY Disk Space is now: OK on fr-wiki-db-precise i-0000023e output: DISK OK [07:12:57] RECOVERY Total Processes is now: OK on fr-wiki-db-precise i-0000023e output: PROCS OK: 88 processes [07:13:04] RECOVERY Disk Space is now: OK on psm-precise i-000002f2 output: DISK OK [07:13:04] RECOVERY SSH is now: OK on bots-sql2 i-000000af output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:13:15] RECOVERY dpkg-check is now: OK on bots-sql2 i-000000af output: All packages OK [07:13:33] PROBLEM Current Load is now: WARNING on etherpad-lite i-000002de output: WARNING - load average: 1.67, 5.41, 5.76 [07:14:16] PROBLEM dpkg-check is now: CRITICAL on build-precise1 i-00000273 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:33] PROBLEM Current Users is now: CRITICAL on build-precise1 i-00000273 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:33] PROBLEM Current Load is now: CRITICAL on build-precise1 i-00000273 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:34] PROBLEM Disk Space is now: CRITICAL on build-precise1 i-00000273 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:34] PROBLEM Free ram is now: CRITICAL on build-precise1 i-00000273 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:45] PROBLEM Total Processes is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:05] PROBLEM Free ram is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:05] PROBLEM Current Load is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:05] PROBLEM Current Users is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:05] PROBLEM dpkg-check is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:28] PROBLEM Current Load is now: WARNING on psm-precise i-000002f2 output: WARNING - load average: 4.23, 5.03, 5.33 [07:15:55] RECOVERY Disk Space is now: OK on incubator-bot2 i-00000252 output: DISK OK [07:15:55] RECOVERY Current Load is now: OK on incubator-bot2 i-00000252 output: OK - load average: 1.45, 2.71, 3.85 [07:15:55] RECOVERY Current Users is now: OK on incubator-bot2 i-00000252 output: USERS OK - 0 users currently logged in [07:15:55] RECOVERY Free ram is now: OK on incubator-bot2 i-00000252 output: OK: 36% free memory [07:15:55] RECOVERY dpkg-check is now: OK on incubator-bot2 i-00000252 output: All packages OK [07:16:56] PROBLEM Current Load is now: WARNING on integration-apache1 i-000002eb output: WARNING - load average: 4.59, 7.13, 6.83 [07:17:07] PROBLEM Current Load is now: WARNING on nova-precise1 i-00000236 output: WARNING - load average: 3.50, 5.57, 5.86 [07:17:40] RECOVERY Current Load is now: OK on wikidata-dev-2 i-00000259 output: OK - load average: 1.66, 2.29, 4.32 [07:17:40] PROBLEM Puppet freshness is now: CRITICAL on maps-test2 i-00000253 output: Puppet has not run in last 20 hours [07:17:41] RECOVERY Current Load is now: OK on etherpad-lite i-000002de output: OK - load average: 0.14, 2.10, 4.22 [07:17:41] RECOVERY Total Processes is now: OK on en-wiki-db-precise i-0000023c output: PROCS OK: 85 processes [07:17:45] PROBLEM Current Load is now: WARNING on aggregator-test1 i-000002bf output: WARNING - load average: 0.87, 7.68, 7.12 [07:17:46] RECOVERY Free ram is now: OK on en-wiki-db-precise i-0000023c output: OK: 78% free memory [07:17:46] RECOVERY Current Load is now: OK on en-wiki-db-precise i-0000023c output: OK - load average: 0.11, 2.92, 4.42 [07:17:46] RECOVERY Current Users is now: OK on en-wiki-db-precise i-0000023c output: USERS OK - 0 users currently logged in [07:17:46] RECOVERY dpkg-check is now: OK on en-wiki-db-precise i-0000023c output: All packages OK [07:17:51] RECOVERY Current Users is now: OK on build-precise1 i-00000273 output: USERS OK - 1 users currently logged in [07:17:57] RECOVERY Disk Space is now: OK on build-precise1 i-00000273 output: DISK OK [07:18:03] PROBLEM Current Load is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:03] PROBLEM Current Users is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:03] PROBLEM Disk Space is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:03] PROBLEM Total Processes is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:08] PROBLEM dpkg-check is now: CRITICAL on deployment-apache30 i-000002d3 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:20:17] RECOVERY Current Load is now: OK on psm-precise i-000002f2 output: OK - load average: 0.07, 1.94, 3.91 [07:21:05] RECOVERY Current Load is now: OK on labs-nfs1 i-0000005d output: OK - load average: 0.37, 2.13, 3.88 [07:21:05] RECOVERY Disk Space is now: OK on incubator-bot0 i-00000296 output: DISK OK [07:21:05] RECOVERY Current Users is now: OK on incubator-bot0 i-00000296 output: USERS OK - 0 users currently logged in [07:21:05] RECOVERY Current Load is now: OK on incubator-bot0 i-00000296 output: OK - load average: 2.79, 3.99, 4.43 [07:21:05] RECOVERY Total Processes is now: OK on incubator-bot0 i-00000296 output: PROCS OK: 87 processes [07:21:10] RECOVERY Free ram is now: OK on incubator-bot0 i-00000296 output: OK: 86% free memory [07:21:10] RECOVERY dpkg-check is now: OK on incubator-bot0 i-00000296 output: All packages OK [07:21:40] RECOVERY Current Load is now: OK on integration-apache1 i-000002eb output: OK - load average: 0.24, 2.72, 4.99 [07:21:40] RECOVERY Current Load is now: OK on nova-precise1 i-00000236 output: OK - load average: 0.10, 2.16, 4.30 [07:22:40] RECOVERY dpkg-check is now: OK on build-precise1 i-00000273 output: All packages OK [07:22:40] RECOVERY Current Load is now: OK on deployment-apache30 i-000002d3 output: OK - load average: 0.44, 3.76, 4.84 [07:22:40] RECOVERY Current Users is now: OK on deployment-apache30 i-000002d3 output: USERS OK - 0 users currently logged in [07:22:40] RECOVERY Disk Space is now: OK on deployment-apache30 i-000002d3 output: DISK OK [07:22:40] RECOVERY Total Processes is now: OK on deployment-apache30 i-000002d3 output: PROCS OK: 119 processes [07:22:45] RECOVERY Current Load is now: OK on build-precise1 i-00000273 output: OK - load average: 0.76, 3.31, 4.62 [07:22:45] RECOVERY Free ram is now: OK on build-precise1 i-00000273 output: OK: 87% free memory [07:22:46] RECOVERY dpkg-check is now: OK on deployment-apache30 i-000002d3 output: All packages OK [07:27:41] RECOVERY Current Load is now: OK on aggregator-test1 i-000002bf output: OK - load average: 0.78, 1.50, 3.96 [07:36:40] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.44, 0.62, 3.40 [07:36:40] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [07:37:40] PROBLEM Free ram is now: CRITICAL on aggregator-test1 i-000002bf output: Critical: 5% free memory [07:41:42] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 2.56, 1.27, 2.85 [07:56:00] PROBLEM dpkg-check is now: CRITICAL on build-precise1 i-00000273 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:56:00] PROBLEM Free ram is now: CRITICAL on build-precise1 i-00000273 output: CHECK_NRPE: Socket timeout after 10 seconds. [08:06:45] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [08:36:56] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 9% free memory [08:36:56] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [08:46:08] PROBLEM Free ram is now: UNKNOWN on psm-precise i-000002f2 output: NRPE: Unable to read output [08:47:36] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [09:07:46] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [09:09:44] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [09:17:04] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [09:21:54] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 10% free memory [09:36:21] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [09:38:03] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [09:56:42] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 199 processes [10:11:59] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [10:21:54] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 208 processes [10:26:54] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 200 processes [10:42:28] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [10:47:42] PROBLEM Free ram is now: WARNING on aggregator-test1 i-000002bf output: Warning: 6% free memory [11:07:15] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 206 processes [11:11:34] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:34] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [11:16:24] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [11:42:35] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [12:12:40] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [12:42:43] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [12:46:47] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [12:51:48] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [12:51:48] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:37] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 7.51, 7.42, 7.57 [13:12:48] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [13:33:02] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:53] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 10% free memory [13:42:56] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [13:55:06] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [14:13:42] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [14:25:24] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [14:26:44] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 0.65, 2.56, 5.00 [14:43:44] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [14:52:27] Really need to fix the 'blocked because of many connection errors' on sql2... keeps breaking things =/ [14:55:28] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [15:17:56] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [15:23:16] PROBLEM Puppet freshness is now: CRITICAL on deployment-transcoding i-00000105 output: Puppet has not run in last 20 hours [15:25:26] PROBLEM Puppet freshness is now: CRITICAL on gerrit i-000000ff output: Puppet has not run in last 20 hours [15:25:57] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [15:32:36] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [15:37:26] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [15:47:56] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [15:56:02] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [16:00:36] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [16:17:26] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 199 processes [16:17:56] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [16:26:06] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [16:26:57] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [16:31:46] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [16:37:29] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 205 processes [16:42:28] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 199 processes [16:47:58] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [16:56:48] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [17:17:58] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [17:18:18] PROBLEM Puppet freshness is now: CRITICAL on maps-test2 i-00000253 output: Puppet has not run in last 20 hours [17:20:58] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [17:26:48] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [17:41:01] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 20% free memory [17:48:01] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [17:54:02] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [17:56:52] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [18:02:07] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [18:06:43] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [18:07:33] PROBLEM Total Processes is now: CRITICAL on aggregator-test1 i-000002bf output: PROCS CRITICAL: 201 processes [18:18:03] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [18:23:23] PROBLEM Puppet freshness is now: CRITICAL on su-fe1 i-000002e5 output: Puppet has not run in last 20 hours [18:26:53] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [18:48:10] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [18:57:20] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [19:17:42] PROBLEM Free ram is now: CRITICAL on wikistats-history-01 i-000002e2 output: CHECK_NRPE: Socket timeout after 10 seconds. [19:18:12] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [19:22:32] PROBLEM Free ram is now: UNKNOWN on wikistats-history-01 i-000002e2 output: NRPE: Unable to read output [19:27:39] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [19:48:19] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [19:58:09] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [20:11:59] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [20:16:50] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [20:19:19] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [20:28:19] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [20:47:35] PROBLEM Total Processes is now: WARNING on aggregator-test1 i-000002bf output: PROCS WARNING: 194 processes [20:49:45] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [20:58:19] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [21:19:46] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [21:28:22] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [21:49:49] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [21:58:27] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [22:01:27] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 18% free memory [22:19:57] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [22:28:27] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [22:49:59] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [22:58:29] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [23:17:23] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [23:20:17] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [23:22:07] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [23:29:57] PROBLEM host: signwriting-ase2 is DOWN address: i-000002fd CRITICAL - Host Unreachable (i-000002fd) [23:51:17] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0)