[00:02:16] PROBLEM Puppet freshness is now: CRITICAL on bots-apache1 i-000000b0 output: Puppet has not run in last 20 hours [00:02:16] PROBLEM Puppet freshness is now: CRITICAL on secondinstance i-0000015b output: Puppet has not run in last 20 hours [00:04:39] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [00:05:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-collector i-000000b7 output: Puppet has not run in last 20 hours [00:08:34] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [00:10:04] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [00:10:04] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [00:12:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-deb i-000002b5 output: Puppet has not run in last 20 hours [00:12:14] PROBLEM Puppet freshness is now: CRITICAL on mwreview-test6 i-000002b9 output: Puppet has not run in last 20 hours [00:12:14] PROBLEM Puppet freshness is now: CRITICAL on queue-wiki1 i-000002b8 output: Puppet has not run in last 20 hours [00:12:14] PROBLEM Puppet freshness is now: CRITICAL on redis1 i-000002b6 output: Puppet has not run in last 20 hours [00:12:14] PROBLEM Puppet freshness is now: CRITICAL on zeromq1 i-000002b7 output: Puppet has not run in last 20 hours [00:13:14] PROBLEM Puppet freshness is now: CRITICAL on shop-analytics-main i-000001e6 output: Puppet has not run in last 20 hours [00:27:08] PROBLEM Puppet freshness is now: CRITICAL on building i-0000014d output: Puppet has not run in last 20 hours [00:32:12] PROBLEM Puppet freshness is now: CRITICAL on swift-fe1 i-000001d2 output: Puppet has not run in last 20 hours [00:35:03] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [00:38:42] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [00:40:32] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [00:40:42] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [00:42:12] PROBLEM Puppet freshness is now: CRITICAL on firstinstance i-0000013e output: Puppet has not run in last 20 hours [00:42:12] PROBLEM Puppet freshness is now: CRITICAL on log1 i-00000239 output: Puppet has not run in last 20 hours [00:45:12] PROBLEM Puppet freshness is now: CRITICAL on pybal-precise i-00000289 output: Puppet has not run in last 20 hours [00:47:12] PROBLEM Puppet freshness is now: CRITICAL on reportcard2 i-000001ea output: Puppet has not run in last 20 hours [01:01:25] PROBLEM Puppet freshness is now: CRITICAL on deployment-imagescaler01 i-0000025a output: Puppet has not run in last 20 hours [01:01:25] PROBLEM Puppet freshness is now: CRITICAL on worker1 i-00000208 output: Puppet has not run in last 20 hours [01:05:10] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [01:08:44] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [01:09:14] PROBLEM Puppet freshness is now: CRITICAL on migration1 i-00000261 output: Puppet has not run in last 20 hours [01:10:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-bits i-00000264 output: Puppet has not run in last 20 hours [01:10:44] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [01:11:44] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [01:18:14] PROBLEM Puppet freshness is now: CRITICAL on catsort-pub i-000001cc output: Puppet has not run in last 20 hours [01:20:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache21 i-0000026d output: Puppet has not run in last 20 hours [01:20:14] PROBLEM Puppet freshness is now: CRITICAL on labs-build1 i-0000006b output: Puppet has not run in last 20 hours [01:22:14] PROBLEM Puppet freshness is now: CRITICAL on bots-sql2 i-000000af output: Puppet has not run in last 20 hours [01:22:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache20 i-0000026c output: Puppet has not run in last 20 hours [01:22:14] PROBLEM Puppet freshness is now: CRITICAL on dumps-2 i-00000257 output: Puppet has not run in last 20 hours [01:24:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-sql i-000000d0 output: Puppet has not run in last 20 hours [01:24:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-wmsearch i-000000e1 output: Puppet has not run in last 20 hours [01:24:14] PROBLEM Puppet freshness is now: CRITICAL on hugglewa-1 i-000001e0 output: Puppet has not run in last 20 hours [01:24:14] PROBLEM Puppet freshness is now: CRITICAL on master i-0000007a output: Puppet has not run in last 20 hours [01:24:14] PROBLEM Puppet freshness is now: CRITICAL on mingledbtest i-00000283 output: Puppet has not run in last 20 hours [01:26:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [01:26:14] PROBLEM Puppet freshness is now: CRITICAL on p-b i-000000ae output: Puppet has not run in last 20 hours [01:27:14] PROBLEM Puppet freshness is now: CRITICAL on feeds i-000000fa output: Puppet has not run in last 20 hours [01:27:14] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot0 i-00000296 output: Puppet has not run in last 20 hours [01:28:14] PROBLEM Puppet freshness is now: CRITICAL on asher1 i-0000003a output: Puppet has not run in last 20 hours [01:28:14] PROBLEM Puppet freshness is now: CRITICAL on pediapress-ocg2 i-00000234 output: Puppet has not run in last 20 hours [01:28:14] PROBLEM Puppet freshness is now: CRITICAL on udp-filter i-000001df output: Puppet has not run in last 20 hours [01:29:14] PROBLEM Puppet freshness is now: CRITICAL on bob i-0000012d output: Puppet has not run in last 20 hours [01:29:14] PROBLEM Puppet freshness is now: CRITICAL on bots-dev i-00000190 output: Puppet has not run in last 20 hours [01:29:14] PROBLEM Puppet freshness is now: CRITICAL on e3 i-00000291 output: Puppet has not run in last 20 hours [01:29:14] PROBLEM Puppet freshness is now: CRITICAL on testing-groupchange i-00000205 output: Puppet has not run in last 20 hours [01:30:14] PROBLEM Puppet freshness is now: CRITICAL on bots-labs i-0000015e output: Puppet has not run in last 20 hours [01:30:14] PROBLEM Puppet freshness is now: CRITICAL on demo-deployment1 i-00000276 output: Puppet has not run in last 20 hours [01:30:14] PROBLEM Puppet freshness is now: CRITICAL on fundraising-db i-0000015c output: Puppet has not run in last 20 hours [01:30:14] PROBLEM Puppet freshness is now: CRITICAL on otrs-jgreen i-0000015a output: Puppet has not run in last 20 hours [01:30:14] PROBLEM Puppet freshness is now: CRITICAL on ve-nodejs i-00000245 output: Puppet has not run in last 20 hours [01:31:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test4 i-000002a2 output: Puppet has not run in last 20 hours [01:31:14] PROBLEM Puppet freshness is now: CRITICAL on incubator-common i-00000254 output: Puppet has not run in last 20 hours [01:31:14] PROBLEM Puppet freshness is now: CRITICAL on varnish i-000001ac output: Puppet has not run in last 20 hours [01:32:15] PROBLEM Puppet freshness is now: CRITICAL on tw-next i-0000027e output: Puppet has not run in last 20 hours [01:33:18] PROBLEM Puppet freshness is now: CRITICAL on bots-nfs i-000000b1 output: Puppet has not run in last 20 hours [01:33:18] PROBLEM Puppet freshness is now: CRITICAL on outreacheval i-0000012e output: Puppet has not run in last 20 hours [01:33:18] PROBLEM Puppet freshness is now: CRITICAL on webserver-lcarr i-00000134 output: Puppet has not run in last 20 hours [01:35:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-jobrunner05 i-0000028c output: Puppet has not run in last 20 hours [01:37:15] PROBLEM Puppet freshness is now: CRITICAL on bots-sql3 i-000000b4 output: Puppet has not run in last 20 hours [01:37:15] PROBLEM Puppet freshness is now: CRITICAL on build-precise1 i-00000273 output: Puppet has not run in last 20 hours [01:37:15] PROBLEM Puppet freshness is now: CRITICAL on en-wiki-db-lucid i-0000023b output: Puppet has not run in last 20 hours [01:37:15] PROBLEM Puppet freshness is now: CRITICAL on exim-test i-00000265 output: Puppet has not run in last 20 hours [01:37:15] PROBLEM Puppet freshness is now: CRITICAL on fr-wiki-db-precise i-0000023e output: Puppet has not run in last 20 hours [01:37:15] PROBLEM Puppet freshness is now: CRITICAL on incubator-apache i-00000211 output: Puppet has not run in last 20 hours [01:37:15] PROBLEM Puppet freshness is now: CRITICAL on labs-realserver i-00000104 output: Puppet has not run in last 20 hours [01:37:16] PROBLEM Puppet freshness is now: CRITICAL on nginx-ffuqua-doom1-3 i-00000196 output: Puppet has not run in last 20 hours [01:37:16] PROBLEM Puppet freshness is now: CRITICAL on pediapress-ocg1 i-00000233 output: Puppet has not run in last 20 hours [01:37:17] PROBLEM Puppet freshness is now: CRITICAL on pediapress-packager i-000001e4 output: Puppet has not run in last 20 hours [01:37:17] PROBLEM Puppet freshness is now: CRITICAL on scribunto i-0000022c output: Puppet has not run in last 20 hours [01:37:18] PROBLEM Puppet freshness is now: CRITICAL on simplewikt i-00000149 output: Puppet has not run in last 20 hours [01:37:18] PROBLEM Puppet freshness is now: CRITICAL on tutorial-mysql i-0000028b output: Puppet has not run in last 20 hours [01:37:19] PROBLEM Puppet freshness is now: CRITICAL on vumi-gw1 i-0000008f output: Puppet has not run in last 20 hours [01:37:43] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [01:38:44] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [01:39:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test3 i-0000025b output: Puppet has not run in last 20 hours [01:39:14] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot1 i-00000251 output: Puppet has not run in last 20 hours [01:39:14] PROBLEM Puppet freshness is now: CRITICAL on pageviews i-000000b2 output: Puppet has not run in last 20 hours [01:40:54] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [01:41:44] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [01:42:14] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [01:43:14] PROBLEM Puppet freshness is now: CRITICAL on bastion-restricted1 i-0000019b output: Puppet has not run in last 20 hours [01:43:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-mc i-0000021b output: Puppet has not run in last 20 hours [01:43:14] PROBLEM Puppet freshness is now: CRITICAL on nova-dev3 i-000000e9 output: Puppet has not run in last 20 hours [01:43:14] PROBLEM Puppet freshness is now: CRITICAL on patchtest i-000000f1 output: Puppet has not run in last 20 hours [01:43:14] PROBLEM Puppet freshness is now: CRITICAL on swift-aux2 i-0000024c output: Puppet has not run in last 20 hours [01:43:14] PROBLEM Puppet freshness is now: CRITICAL on vumi i-000001e5 output: Puppet has not run in last 20 hours [01:44:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-squid i-000000dc output: Puppet has not run in last 20 hours [01:44:14] PROBLEM Puppet freshness is now: CRITICAL on embed-sandbox i-000000d1 output: Puppet has not run in last 20 hours [01:44:14] PROBLEM Puppet freshness is now: CRITICAL on opengrok-web i-000001e1 output: Puppet has not run in last 20 hours [01:44:14] PROBLEM Puppet freshness is now: CRITICAL on swift-aux1 i-0000024b output: Puppet has not run in last 20 hours [01:44:14] PROBLEM Puppet freshness is now: CRITICAL on vivek-puppet i-000000ca output: Puppet has not run in last 20 hours [01:46:14] PROBLEM Puppet freshness is now: CRITICAL on dev-solr i-00000152 output: Puppet has not run in last 20 hours [01:47:15] PROBLEM Puppet freshness is now: CRITICAL on demo-web2 i-00000285 output: Puppet has not run in last 20 hours [01:47:15] PROBLEM Puppet freshness is now: CRITICAL on en-wiki-db-precise i-0000023c output: Puppet has not run in last 20 hours [01:47:15] PROBLEM Puppet freshness is now: CRITICAL on ubuntu1-pgehres i-000000fb output: Puppet has not run in last 20 hours [01:48:14] PROBLEM Puppet freshness is now: CRITICAL on maps-test2 i-00000253 output: Puppet has not run in last 20 hours [01:49:14] PROBLEM Puppet freshness is now: CRITICAL on bots-1 i-000000a9 output: Puppet has not run in last 20 hours [01:50:14] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 72 MB (5% inode=58%): [01:51:14] PROBLEM Puppet freshness is now: CRITICAL on build1 i-000002b3 output: Puppet has not run in last 20 hours [01:51:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-nfs-memc i-000000d7 output: Puppet has not run in last 20 hours [01:52:15] PROBLEM Puppet freshness is now: CRITICAL on hugglewa-db i-00000188 output: Puppet has not run in last 20 hours [01:52:15] PROBLEM Puppet freshness is now: CRITICAL on maps-test3 i-0000028f output: Puppet has not run in last 20 hours [01:52:15] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap2 i-00000238 output: Puppet has not run in last 20 hours [01:53:14] PROBLEM Puppet freshness is now: CRITICAL on translation-memory-1 i-0000013a output: Puppet has not run in last 20 hours [02:08:44] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [02:08:44] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [02:10:54] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [02:11:44] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [02:39:35] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [02:39:35] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [02:40:54] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [02:41:44] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [02:44:44] RECOVERY Current Load is now: OK on mobile-wlm i-000002bc output: OK - load average: 0.48, 0.43, 0.54 [02:45:04] RECOVERY Current Users is now: OK on mobile-wlm i-000002bc output: USERS OK - 0 users currently logged in [02:45:44] RECOVERY Free ram is now: OK on mobile-wlm i-000002bc output: OK: 83% free memory [02:46:34] RECOVERY Disk Space is now: OK on mobile-wlm i-000002bc output: DISK OK [02:48:34] RECOVERY Total Processes is now: OK on mobile-wlm i-000002bc output: PROCS OK: 91 processes [02:48:44] RECOVERY dpkg-check is now: OK on mobile-wlm i-000002bc output: All packages OK [02:53:40] PROBLEM dpkg-check is now: CRITICAL on nova-daas-1 i-000000e7 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:58:40] RECOVERY dpkg-check is now: OK on nova-daas-1 i-000000e7 output: All packages OK [03:01:51] PROBLEM Disk Space is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:01:51] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:01:51] PROBLEM SSH is now: CRITICAL on bots-sql2 i-000000af output: CRITICAL - Socket timeout after 10 seconds [03:01:51] PROBLEM Current Users is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:02:57] PROBLEM Total Processes is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:03:02] PROBLEM Current Users is now: CRITICAL on ganglia-test5 i-000002a7 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:03:16] PROBLEM Total Processes is now: CRITICAL on ganglia-test5 i-000002a7 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:04:21] 06/03/2012 - 03:04:21 - Updating keys for laner at /export/home/deployment-prep/laner [03:04:28] PROBLEM Disk Space is now: CRITICAL on ganglia-test5 i-000002a7 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:04:28] PROBLEM Free ram is now: CRITICAL on ganglia-test5 i-000002a7 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:04:28] PROBLEM dpkg-check is now: CRITICAL on ganglia-test5 i-000002a7 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:05:01] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 7.89, 8.69, 5.50 [03:06:34] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 4.75, 10.03, 7.25 [03:07:01] PROBLEM Total Processes is now: CRITICAL on ganglia-test6 i-000002af output: CHECK_NRPE: Socket timeout after 10 seconds. [03:07:06] PROBLEM dpkg-check is now: CRITICAL on ganglia-test6 i-000002af output: CHECK_NRPE: Socket timeout after 10 seconds. [03:07:06] PROBLEM Free ram is now: CRITICAL on ganglia-test6 i-000002af output: CHECK_NRPE: Socket timeout after 10 seconds. [03:07:15] RECOVERY Disk Space is now: OK on rds i-00000207 output: DISK OK [03:07:15] RECOVERY Current Load is now: OK on rds i-00000207 output: OK - load average: 10.01, 7.60, 4.36 [03:07:15] RECOVERY Current Users is now: OK on rds i-00000207 output: USERS OK - 0 users currently logged in [03:10:02] RECOVERY Disk Space is now: OK on ganglia-test5 i-000002a7 output: DISK OK [03:10:02] RECOVERY Free ram is now: OK on ganglia-test5 i-000002a7 output: OK: 84% free memory [03:10:02] RECOVERY dpkg-check is now: OK on ganglia-test5 i-000002a7 output: All packages OK [03:10:06] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [03:10:42] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [03:11:23] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [03:11:23] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [03:11:23] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [03:12:27] RECOVERY SSH is now: OK on bots-sql2 i-000000af output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [03:13:53] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [03:13:53] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [03:13:58] RECOVERY Total Processes is now: OK on ganglia-test5 i-000002a7 output: PROCS OK: 181 processes [03:14:03] RECOVERY Current Users is now: OK on ganglia-test5 i-000002a7 output: USERS OK - 0 users currently logged in [03:14:03] RECOVERY Total Processes is now: OK on rds i-00000207 output: PROCS OK: 75 processes [03:15:18] RECOVERY Total Processes is now: OK on ganglia-test6 i-000002af output: PROCS OK: 80 processes [03:15:28] RECOVERY dpkg-check is now: OK on ganglia-test6 i-000002af output: All packages OK [03:15:28] RECOVERY Free ram is now: OK on ganglia-test6 i-000002af output: OK: 91% free memory [03:16:08] RECOVERY Current Users is now: OK on upload-wizard i-0000021c output: USERS OK - 0 users currently logged in [03:16:08] RECOVERY Disk Space is now: OK on upload-wizard i-0000021c output: DISK OK [03:16:13] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 94% free memory [03:19:38] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 1.16, 3.29, 4.83 [03:25:21] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.77, 1.06, 3.57 [03:30:18] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 1.27, 0.87, 2.76 [03:38:18] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 16% free memory [03:40:28] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [03:42:58] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [03:43:56] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [03:43:56] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [03:44:51] PROBLEM Puppet freshness is now: CRITICAL on blamemaps-m1small i-000002a1 output: Puppet has not run in last 20 hours [03:56:53] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:58:23] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 3% free memory [04:02:00] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [04:03:30] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 96% free memory [04:10:10] PROBLEM Puppet freshness is now: CRITICAL on mobile-testing i-00000271 output: Puppet has not run in last 20 hours [04:11:01] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [04:12:10] PROBLEM Puppet freshness is now: CRITICAL on precise-test i-00000231 output: Puppet has not run in last 20 hours [04:14:10] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [04:14:10] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [04:14:10] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [04:15:20] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 17% free memory [04:17:00] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 3% free memory [04:20:26] PROBLEM Puppet freshness is now: CRITICAL on bots-3 i-000000e5 output: Puppet has not run in last 20 hours [04:22:03] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 96% free memory [04:23:13] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test5 i-000002a7 output: Puppet has not run in last 20 hours [04:23:13] PROBLEM Puppet freshness is now: CRITICAL on mwreview i-000002ae output: Puppet has not run in last 20 hours [04:24:13] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot2 i-00000252 output: Puppet has not run in last 20 hours [04:25:13] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test6 i-000002af output: Puppet has not run in last 20 hours [04:26:13] PROBLEM Puppet freshness is now: CRITICAL on localpuppet1 i-0000020b output: Puppet has not run in last 20 hours [04:27:03] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:33:13] PROBLEM Puppet freshness is now: CRITICAL on maps-tilemill1 i-00000294 output: Puppet has not run in last 20 hours [04:37:03] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:40:23] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 3% free memory [04:41:03] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [04:44:13] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [04:44:13] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [04:44:13] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [04:45:23] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 97% free memory [04:50:13] PROBLEM Puppet freshness is now: CRITICAL on mailman-01 i-00000235 output: Puppet has not run in last 20 hours [04:53:56] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [05:01:53] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 72 MB (5% inode=58%): [05:11:03] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [05:14:13] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [05:14:13] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [05:14:13] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [05:41:03] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [05:44:13] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [05:44:13] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [05:44:13] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [06:11:03] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [06:14:13] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [06:14:13] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [06:14:13] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [06:21:13] PROBLEM Puppet freshness is now: CRITICAL on bots-cb i-0000009e output: Puppet has not run in last 20 hours [06:32:29] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg2 i-00000234 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:32:29] PROBLEM Disk Space is now: CRITICAL on pediapress-ocg2 i-00000234 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:34:34] PROBLEM Total Processes is now: CRITICAL on e3 i-00000291 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:37:19] RECOVERY Disk Space is now: OK on pediapress-ocg2 i-00000234 output: DISK OK [06:37:19] RECOVERY dpkg-check is now: OK on pediapress-ocg2 i-00000234 output: All packages OK [06:38:12] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 4.24, 4.62, 2.64 [06:38:12] RECOVERY Total Processes is now: OK on e3 i-00000291 output: PROCS OK: 94 processes [06:38:17] PROBLEM dpkg-check is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:17] PROBLEM Current Load is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:18] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:18] PROBLEM Current Users is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:18] PROBLEM Disk Space is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:18] PROBLEM Current Users is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:52] PROBLEM Disk Space is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:38:52] PROBLEM Free ram is now: CRITICAL on nova-precise1 i-00000236 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:41:19] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [06:41:29] PROBLEM Current Load is now: CRITICAL on ganglia-test4 i-000002a2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:41:29] PROBLEM Total Processes is now: CRITICAL on ganglia-test4 i-000002a2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:06] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 1.17, 2.56, 2.29 [06:43:07] RECOVERY Current Load is now: OK on nova-precise1 i-00000236 output: OK - load average: 8.54, 7.58, 4.23 [06:43:07] RECOVERY Current Users is now: OK on nova-precise1 i-00000236 output: USERS OK - 0 users currently logged in [06:43:07] RECOVERY dpkg-check is now: OK on nova-precise1 i-00000236 output: All packages OK [06:43:22] PROBLEM Current Users is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:22] PROBLEM dpkg-check is now: CRITICAL on deployment-jobrunner05 i-0000028c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:36] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 7.57, 7.39, 5.50 [06:43:36] RECOVERY Disk Space is now: OK on nova-precise1 i-00000236 output: DISK OK [06:43:36] RECOVERY Free ram is now: OK on nova-precise1 i-00000236 output: OK: 76% free memory [06:43:46] PROBLEM Disk Space is now: CRITICAL on ganglia-test4 i-000002a2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:46] PROBLEM dpkg-check is now: CRITICAL on ganglia-test4 i-000002a2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:07] PROBLEM Current Load is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:07] PROBLEM Total Processes is now: CRITICAL on pediapress-ocg1 i-00000233 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:32] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [06:45:32] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [06:45:32] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [06:46:18] PROBLEM Total Processes is now: CRITICAL on e3 i-00000291 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:56] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [06:49:06] PROBLEM Total Processes is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:14] PROBLEM Free ram is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:14] PROBLEM Current Load is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:14] PROBLEM HTTP is now: CRITICAL on mailman-01 i-00000235 output: Connection refused [06:49:14] RECOVERY Disk Space is now: OK on pediapress-ocg1 i-00000233 output: DISK OK [06:49:14] RECOVERY Current Users is now: OK on pediapress-ocg1 i-00000233 output: USERS OK - 0 users currently logged in [06:49:14] RECOVERY dpkg-check is now: OK on pediapress-ocg1 i-00000233 output: All packages OK [06:49:14] RECOVERY Current Load is now: OK on pediapress-ocg1 i-00000233 output: OK - load average: 1.97, 3.95, 3.31 [06:49:15] RECOVERY Total Processes is now: OK on pediapress-ocg1 i-00000233 output: PROCS OK: 90 processes [06:49:28] PROBLEM dpkg-check is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:28] PROBLEM Free ram is now: CRITICAL on deployment-jobrunner05 i-0000028c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:28] PROBLEM Disk Space is now: CRITICAL on incubator-bot0 i-00000296 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:30] PROBLEM dpkg-check is now: CRITICAL on e3 i-00000291 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:55] PROBLEM Free ram is now: CRITICAL on zeromq1 i-000002b7 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:56] PROBLEM Free ram is now: CRITICAL on e3 i-00000291 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:56] PROBLEM Current Users is now: CRITICAL on ganglia-test4 i-000002a2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:54:26] RECOVERY Disk Space is now: OK on deployment-transcoding i-00000105 output: DISK OK [06:56:18] PROBLEM Total Processes is now: CRITICAL on zeromq1 i-000002b7 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:38] PROBLEM Free ram is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:38] PROBLEM Free ram is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:38] PROBLEM dpkg-check is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:56:38] PROBLEM Disk Space is now: CRITICAL on maps-test2 i-00000253 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:11] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 3.85, 7.58, 5.91 [06:57:11] RECOVERY dpkg-check is now: OK on e3 i-00000291 output: All packages OK [06:57:11] RECOVERY Free ram is now: OK on e3 i-00000291 output: OK: 91% free memory [06:57:14] RECOVERY Disk Space is now: OK on ganglia-test4 i-000002a2 output: DISK OK [06:57:14] RECOVERY Current Users is now: OK on ganglia-test4 i-000002a2 output: USERS OK - 0 users currently logged in [06:57:14] RECOVERY dpkg-check is now: OK on ganglia-test4 i-000002a2 output: All packages OK [06:58:36] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:17] PROBLEM Disk Space is now: CRITICAL on pediapress-ocg2 i-00000234 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:02:27] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg2 i-00000234 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:02:27] PROBLEM Disk Space is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:02:27] PROBLEM Current Users is now: CRITICAL on en-wiki-db-precise i-0000023c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:11] PROBLEM Current Load is now: WARNING on deployment-jobrunner05 i-0000028c output: WARNING - load average: 6.59, 9.63, 9.04 [07:04:13] PROBLEM Current Load is now: WARNING on ganglia-test4 i-000002a2 output: WARNING - load average: 5.50, 6.13, 5.45 [07:04:14] RECOVERY Total Processes is now: OK on ganglia-test4 i-000002a2 output: PROCS OK: 198 processes [07:04:23] RECOVERY Free ram is now: OK on en-wiki-db-precise i-0000023c output: OK: 84% free memory [07:04:24] RECOVERY Free ram is now: OK on maps-test2 i-00000253 output: OK: 89% free memory [07:04:24] RECOVERY dpkg-check is now: OK on maps-test2 i-00000253 output: All packages OK [07:04:29] RECOVERY Disk Space is now: OK on maps-test2 i-00000253 output: DISK OK [07:04:29] RECOVERY Free ram is now: OK on deployment-jobrunner05 i-0000028c output: OK: 87% free memory [07:04:29] RECOVERY Current Users is now: OK on incubator-bot0 i-00000296 output: USERS OK - 0 users currently logged in [07:04:29] RECOVERY dpkg-check is now: OK on deployment-jobrunner05 i-0000028c output: All packages OK [07:07:20] RECOVERY Free ram is now: OK on zeromq1 i-000002b7 output: OK: 88% free memory [07:07:26] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 5.26, 7.28, 7.96 [07:08:47] RECOVERY Disk Space is now: OK on en-wiki-db-precise i-0000023c output: DISK OK [07:08:48] RECOVERY Current Users is now: OK on en-wiki-db-precise i-0000023c output: USERS OK - 0 users currently logged in [07:08:53] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 77 MB (5% inode=57%): [07:09:14] RECOVERY Current Load is now: OK on ganglia-test4 i-000002a2 output: OK - load average: 0.20, 2.66, 4.18 [07:09:19] PROBLEM Free ram is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:09:19] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:09:19] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:09:19] PROBLEM Current Load is now: WARNING on pybal-precise i-00000289 output: WARNING - load average: 7.28, 9.00, 6.65 [07:09:24] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:09:29] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:09:29] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:10:09] PROBLEM Total Processes is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:10:13] PROBLEM Disk Space is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:29] RECOVERY Total Processes is now: OK on zeromq1 i-000002b7 output: PROCS OK: 82 processes [07:11:40] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [07:11:45] PROBLEM dpkg-check is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:05] PROBLEM Total Processes is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:17] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:17] PROBLEM dpkg-check is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:58] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 9.52, 17.20, 11.64 [07:13:19] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:12] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:12] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:13] RECOVERY Free ram is now: OK on pybal-precise i-00000289 output: OK: 79% free memory [07:14:13] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:13] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:58] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:47] PROBLEM Current Load is now: WARNING on upload-wizard i-0000021c output: WARNING - load average: 4.70, 6.77, 5.89 [07:15:47] RECOVERY dpkg-check is now: OK on bots-sql2 i-000000af output: All packages OK [07:15:47] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [07:15:47] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [07:15:47] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [07:17:53] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 76 processes [07:18:29] RECOVERY Current Load is now: OK on worker1 i-00000208 output: OK - load average: 0.46, 3.83, 4.57 [07:18:29] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [07:18:29] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [07:18:29] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 95% free memory [07:18:29] RECOVERY Disk Space is now: OK on upload-wizard i-0000021c output: DISK OK [07:18:29] RECOVERY Current Users is now: OK on upload-wizard i-0000021c output: USERS OK - 0 users currently logged in [07:18:30] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 94% free memory [07:18:30] RECOVERY Current Load is now: OK on pybal-precise i-00000289 output: OK - load average: 0.26, 2.64, 4.56 [07:18:31] RECOVERY Total Processes is now: OK on upload-wizard i-0000021c output: PROCS OK: 83 processes [07:18:35] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 5.08, 11.91, 14.24 [07:18:37] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [07:18:37] RECOVERY Disk Space is now: OK on mobile-testing i-00000271 output: DISK OK [07:18:38] PROBLEM Current Load is now: WARNING on ganglia-test5 i-000002a7 output: WARNING - load average: 5.42, 9.84, 9.14 [07:19:25] PROBLEM Current Load is now: WARNING on maps-tilemill1 i-00000294 output: WARNING - load average: 0.38, 3.29, 5.03 [07:20:28] RECOVERY Current Load is now: OK on upload-wizard i-0000021c output: OK - load average: 0.21, 3.02, 4.54 [07:20:28] RECOVERY Total Processes is now: OK on mobile-testing i-00000271 output: PROCS OK: 131 processes [07:20:33] RECOVERY Free ram is now: OK on mobile-testing i-00000271 output: OK: 91% free memory [07:20:33] RECOVERY dpkg-check is now: OK on mobile-testing i-00000271 output: All packages OK [07:20:48] PROBLEM Current Load is now: WARNING on jenkins2 i-00000102 output: WARNING - load average: 0.70, 6.07, 6.89 [07:23:32] O_o [07:24:18] RECOVERY Current Load is now: OK on maps-tilemill1 i-00000294 output: OK - load average: 0.22, 1.35, 3.70 [07:25:48] RECOVERY Current Load is now: OK on jenkins2 i-00000102 output: OK - load average: 0.00, 2.23, 4.98 [07:28:28] RECOVERY Current Load is now: OK on ganglia-test5 i-000002a7 output: OK - load average: 0.02, 1.45, 4.93 [07:28:28] RECOVERY Current Load is now: OK on deployment-jobrunner05 i-0000028c output: OK - load average: 3.12, 3.47, 4.59 [07:32:28] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.17, 0.58, 3.74 [07:38:28] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 0.93, 0.99, 4.48 [07:43:28] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [07:47:48] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [07:48:28] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [07:48:28] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [07:54:08] PROBLEM Puppet freshness is now: CRITICAL on ipv6test1 i-00000282 output: Puppet has not run in last 20 hours [07:56:08] PROBLEM Puppet freshness is now: CRITICAL on rds i-00000207 output: Puppet has not run in last 20 hours [07:56:08] PROBLEM Puppet freshness is now: CRITICAL on upload-wizard i-0000021c output: Puppet has not run in last 20 hours [08:13:28] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [08:18:28] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [08:18:28] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [08:18:28] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [08:43:28] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [08:45:05] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Husky link https://www.mediawiki.org/w/index.php?diff=546063 edit summary: What about having the URL in this document somewhere? :) [08:48:28] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [08:48:28] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [08:48:28] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [09:08:08] PROBLEM Puppet freshness is now: CRITICAL on cn-wiki-db-lucid i-00000241 output: Puppet has not run in last 20 hours [09:13:28] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [09:15:08] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-1 i-0000020c output: Puppet has not run in last 20 hours [09:16:08] PROBLEM Puppet freshness is now: CRITICAL on demo-mysql1 i-00000256 output: Puppet has not run in last 20 hours [09:18:08] PROBLEM Puppet freshness is now: CRITICAL on dumps-1 i-00000170 output: Puppet has not run in last 20 hours [09:18:08] PROBLEM Puppet freshness is now: CRITICAL on wikistream-1 i-0000016e output: Puppet has not run in last 20 hours [09:18:28] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [09:18:28] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [09:18:28] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [09:19:08] PROBLEM Puppet freshness is now: CRITICAL on swift-be3 i-000001c9 output: Puppet has not run in last 20 hours [09:23:08] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-upload i-00000263 output: Puppet has not run in last 20 hours [09:23:08] PROBLEM Puppet freshness is now: CRITICAL on publicdata-administration i-0000019e output: Puppet has not run in last 20 hours [09:23:08] PROBLEM Puppet freshness is now: CRITICAL on resourceloader2-apache i-000001d7 output: Puppet has not run in last 20 hours [09:23:08] PROBLEM Puppet freshness is now: CRITICAL on test2 i-0000013c output: Puppet has not run in last 20 hours [09:23:08] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-3 i-00000225 output: Puppet has not run in last 20 hours [09:24:08] PROBLEM Puppet freshness is now: CRITICAL on venus i-000000ea output: Puppet has not run in last 20 hours [09:24:08] PROBLEM Puppet freshness is now: CRITICAL on wmde-test i-000002ad output: Puppet has not run in last 20 hours [09:25:08] PROBLEM Puppet freshness is now: CRITICAL on wikisource-web i-000000fe output: Puppet has not run in last 20 hours [09:30:08] PROBLEM Puppet freshness is now: CRITICAL on bots-sql1 i-000000b5 output: Puppet has not run in last 20 hours [09:30:08] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [09:31:08] PROBLEM Puppet freshness is now: CRITICAL on bastion1 i-000000ba output: Puppet has not run in last 20 hours [09:31:08] PROBLEM Puppet freshness is now: CRITICAL on demo-web1 i-00000255 output: Puppet has not run in last 20 hours [09:31:08] PROBLEM Puppet freshness is now: CRITICAL on kripke i-00000268 output: Puppet has not run in last 20 hours [09:31:08] PROBLEM Puppet freshness is now: CRITICAL on labs-nfs1 i-0000005d output: Puppet has not run in last 20 hours [09:34:08]