[00:42:09] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [02:44:19] 05/13/2012 - 02:44:19 - Updating keys for laner [02:51:19] 05/13/2012 - 02:51:19 - Updating keys for laner [02:55:19] 05/13/2012 - 02:55:19 - Updating keys for laner [03:04:22] 05/13/2012 - 03:04:22 - Updating keys for laner [03:22:39] PROBLEM Disk Space is now: CRITICAL on bz-dev i-000001db output: DISK CRITICAL - free space: / 24 MB (1% inode=43%): [03:24:09] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [03:27:39] PROBLEM Disk Space is now: WARNING on bz-dev i-000001db output: DISK WARNING - free space: / 50 MB (3% inode=43%): [03:44:52] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [03:53:03] 05/13/2012 - 03:53:02 - Updating keys for laner at /export/home/testlabs/laner [03:53:03] 05/13/2012 - 03:53:03 - Updating keys for laner at /export/home/juju/laner [03:53:04] 05/13/2012 - 03:53:04 - Updating keys for laner at /export/home/etherpad/laner [03:53:05] 05/13/2012 - 03:53:05 - Updating keys for laner at /export/home/gluster/laner [03:53:06] 05/13/2012 - 03:53:05 - Updating keys for laner at /export/home/mobile-sms/laner [03:53:06] 05/13/2012 - 03:53:06 - Updating keys for laner at /export/home/pediapress/laner [03:53:07] 05/13/2012 - 03:53:07 - Updating keys for laner at /export/home/openstack/laner [03:53:07] 05/13/2012 - 03:53:07 - Updating keys for laner at /export/home/puppet/laner [03:53:08] 05/13/2012 - 03:53:08 - Updating keys for laner at /export/home/membase/laner [03:53:08] 05/13/2012 - 03:53:08 - Updating keys for laner at /export/home/turnkey-mediawiki/laner [03:53:09] 05/13/2012 - 03:53:09 - Updating keys for laner at /export/home/testswarm/laner [03:53:10] 05/13/2012 - 03:53:10 - Updating keys for laner at /export/home/bots/laner [03:53:10] 05/13/2012 - 03:53:10 - Updating keys for laner at /export/home/nagios/laner [03:53:11] 05/13/2012 - 03:53:11 - Updating keys for laner at /export/home/reportcard/laner [03:53:11] 05/13/2012 - 03:53:11 - Updating keys for laner at /export/home/mobile/laner [03:53:12] 05/13/2012 - 03:53:12 - Updating keys for laner at /export/home/storage/laner [03:53:13] 05/13/2012 - 03:53:13 - Updating keys for laner at /export/home/bastion/laner [03:53:14] 05/13/2012 - 03:53:14 - Updating keys for laner at /export/home/mediahandler/laner [03:53:15] 05/13/2012 - 03:53:14 - Updating keys for laner at /export/home/sugarcrm/laner [03:53:15] 05/13/2012 - 03:53:15 - Updating keys for laner at /export/home/ganglia/laner [03:53:16] 05/13/2012 - 03:53:16 - Updating keys for laner at /export/home/gerrit/laner [03:53:17] 05/13/2012 - 03:53:17 - Updating keys for laner at /export/home/deployment-prep/laner [03:53:18] 05/13/2012 - 03:53:18 - Updating keys for laner at /export/home/mobile-stats/laner [03:53:19] 05/13/2012 - 03:53:19 - Updating keys for laner at /export/home/nginx/laner [03:53:19] 05/13/2012 - 03:53:19 - Updating keys for laner at /export/home/patchtest/laner [03:53:20] 05/13/2012 - 03:53:20 - Updating keys for laner at /export/home/selenium/laner [03:53:20] 05/13/2012 - 03:53:20 - Updating keys for laner at /export/home/deployment-prepbackup/laner [03:53:21] 05/13/2012 - 03:53:20 - Updating keys for laner at /export/home/feeds/laner [03:53:32] 05/13/2012 - 03:53:32 - Updating keys for laner at /export/home/publicdata/laner [03:53:34] 05/13/2012 - 03:53:33 - Updating keys for laner at /export/home/shop-analytics/laner [03:53:34] 05/13/2012 - 03:53:34 - Updating keys for laner at /export/home/category-sorting/laner [03:53:35] 05/13/2012 - 03:53:35 - Updating keys for laner at /export/home/wikidata-dev/laner [03:53:38] 05/13/2012 - 03:53:37 - Updating keys for laner at /export/home/opengrok/laner [03:53:38] 05/13/2012 - 03:53:38 - Updating keys for laner at /export/home/gareth/laner [03:53:39] 05/13/2012 - 03:53:38 - Updating keys for laner at /export/home/analytics/laner [03:53:39] 05/13/2012 - 03:53:39 - Updating keys for laner at /export/home/syslog-collection/laner [03:53:40] 05/13/2012 - 03:53:40 - Updating keys for laner at /export/home/mailman/laner [03:53:42] 05/13/2012 - 03:53:41 - Updating keys for laner at /export/home/demo/laner [03:53:42] 05/13/2012 - 03:53:42 - Updating keys for laner at /export/home/wiki-migration/laner [03:53:47] heh [03:54:32] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 15% free memory [03:56:32] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 17% free memory [03:59:32] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [04:04:52] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:09:32] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [04:09:52] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:16:32] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [04:19:32] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:21:32] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:24:42] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 4% free memory [04:29:32] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [05:02:42] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 10% free memory [05:07:43] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [05:16:32] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [05:21:22] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 12% free memory [06:40:02] PROBLEM HTTP is now: CRITICAL on mailman-01 i-00000235 output: Connection refused [06:41:52] PROBLEM Total Processes is now: CRITICAL on fr-wiki-db-precise i-0000023e output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:34] PROBLEM Current Load is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:34] PROBLEM dpkg-check is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:44] RECOVERY Total Processes is now: OK on fr-wiki-db-precise i-0000023e output: PROCS OK: 102 processes [06:52:04] PROBLEM Current Users is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:04] PROBLEM Total Processes is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:32] PROBLEM Current Users is now: CRITICAL on nova-essex-test i-000001f9 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:32] PROBLEM Total Processes is now: CRITICAL on nova-essex-test i-000001f9 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:40] PROBLEM Current Users is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:02:12] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:02:34] PROBLEM Current Load is now: CRITICAL on pediapress-ocg2 i-00000234 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:28] RECOVERY Current Users is now: OK on pediapress-ocg1 i-00000233 output: USERS OK - 0 users currently logged in [07:03:28] RECOVERY Total Processes is now: OK on nova-precise1 i-00000236 output: PROCS OK: 139 processes [07:03:59] RECOVERY Current Users is now: OK on nova-precise1 i-00000236 output: USERS OK - 0 users currently logged in [07:05:20] PROBLEM Current Load is now: WARNING on nova-precise1 i-00000236 output: WARNING - load average: 7.92, 10.20, 9.94 [07:05:21] RECOVERY dpkg-check is now: OK on nova-precise1 i-00000236 output: All packages OK [07:07:13] RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK [07:07:56] PROBLEM Current Load is now: WARNING on pediapress-ocg2 i-00000234 output: WARNING - load average: 4.06, 7.77, 6.40 [07:07:56] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 5.24, 6.79, 6.22 [07:07:56] PROBLEM Current Load is now: WARNING on labs-nfs1 i-0000005d output: WARNING - load average: 0.88, 3.13, 5.15 [07:07:56] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 0.76, 7.24, 12.25 [07:07:56] PROBLEM Current Load is now: WARNING on nova-essex-test i-000001f9 output: WARNING - load average: 4.12, 6.17, 6.85 [07:08:28] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 3.80, 5.52, 5.78 [07:08:28] PROBLEM Current Load is now: WARNING on maps-test2 i-00000253 output: WARNING - load average: 4.19, 6.69, 5.54 [07:09:00] PROBLEM Disk Space is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:09:00] RECOVERY Current Users is now: OK on nova-essex-test i-000001f9 output: USERS OK - 0 users currently logged in [07:09:00] RECOVERY Total Processes is now: OK on nova-essex-test i-000001f9 output: PROCS OK: 123 processes [07:09:30] PROBLEM Current Load is now: WARNING on nova-production1 i-0000007b output: WARNING - load average: 1.87, 4.99, 5.06 [07:09:30] PROBLEM Current Load is now: WARNING on incubator-bot2 i-00000252 output: WARNING - load average: 2.27, 5.04, 5.78 [07:12:50] RECOVERY Current Load is now: OK on pediapress-ocg2 i-00000234 output: OK - load average: 0.27, 3.43, 4.93 [07:12:50] RECOVERY Current Load is now: OK on labs-nfs1 i-0000005d output: OK - load average: 0.28, 1.39, 3.83 [07:12:50] RECOVERY dpkg-check is now: OK on pediapress-ocg1 i-00000233 output: All packages OK [07:13:30] RECOVERY Current Load is now: OK on maps-test2 i-00000253 output: OK - load average: 0.24, 2.90, 4.25 [07:13:50] RECOVERY Disk Space is now: OK on ve-nodejs i-00000245 output: DISK OK [07:14:30] RECOVERY Current Load is now: OK on nova-production1 i-0000007b output: OK - load average: 1.22, 2.33, 3.86 [07:14:30] RECOVERY Current Load is now: OK on incubator-bot2 i-00000252 output: OK - load average: 0.50, 2.17, 4.32 [07:14:50] PROBLEM Disk Space is now: WARNING on deployment-feed i-00000118 output: DISK WARNING - free space: / 77 MB (5% inode=40%): [07:17:50] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.32, 1.33, 3.54 [07:17:50] RECOVERY Current Load is now: OK on nova-essex-test i-000001f9 output: OK - load average: 0.14, 1.21, 3.91 [07:18:30] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 3.08, 3.49, 4.57 [07:20:10] RECOVERY Current Load is now: OK on nova-precise1 i-00000236 output: OK - load average: 0.06, 1.05, 4.56 [07:22:50] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.13, 0.75, 2.68 [07:22:50] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.06, 0.51, 4.76 [08:14:44] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [09:10:07] RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK [09:18:07] PROBLEM Disk Space is now: WARNING on deployment-feed i-00000118 output: DISK WARNING - free space: / 77 MB (5% inode=40%): [10:42:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [13:24:16] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [14:29:47] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [16:50:47] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 0.95, 8.49, 6.16 [17:00:47] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.37, 2.08, 4.20 [18:39:42] !log bots petrb: patching wm-bot [18:39:48] Logged the message, Master [18:55:41] @help suppress-on [18:55:41] Info for suppress-on: Disable output to channel [19:47:20] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 3615 MB (20% inode=80%): [20:02:22] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [20:09:52] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [20:10:22] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 3591 MB (20% inode=80%): [20:43:12] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [20:44:52] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [21:12:52] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [22:44:23] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 3509 MB (20% inode=80%): [23:25:13] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [23:29:23] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [23:37:23] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 3480 MB (20% inode=80%):