[00:00:33] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [00:01:03] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [00:01:03] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [00:19:13] PROBLEM Puppet freshness is now: CRITICAL on bots-cb i-0000009e output: Puppet has not run in last 20 hours [00:26:03] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [00:30:33] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [00:31:03] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [00:31:03] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [00:56:03] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [01:00:33] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [01:01:03] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [01:01:03] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [01:26:03] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [01:30:33] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [01:31:03] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [01:31:03] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [01:53:13] PROBLEM Puppet freshness is now: CRITICAL on ipv6test1 i-00000282 output: Puppet has not run in last 20 hours [01:55:13] PROBLEM Puppet freshness is now: CRITICAL on rds i-00000207 output: Puppet has not run in last 20 hours [01:55:13] PROBLEM Puppet freshness is now: CRITICAL on upload-wizard i-0000021c output: Puppet has not run in last 20 hours [01:56:03] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [02:00:33] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [02:01:03] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [02:01:03] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [02:26:03] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [02:30:33] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [02:31:03] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [02:31:03] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [02:44:21] 06/02/2012 - 02:44:20 - Updating keys for laner at /export/home/deployment-prep/laner [02:56:06] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [02:56:16] RECOVERY Total Processes is now: OK on redis1 i-000002b6 output: PROCS OK: 87 processes [02:56:26] RECOVERY Free ram is now: OK on redis1 i-000002b6 output: OK: 88% free memory [02:56:56] RECOVERY dpkg-check is now: OK on redis1 i-000002b6 output: All packages OK [02:58:06] RECOVERY Current Load is now: OK on redis1 i-000002b6 output: OK - load average: 0.09, 0.13, 0.09 [02:58:47] RECOVERY Current Users is now: OK on redis1 i-000002b6 output: USERS OK - 1 users currently logged in [02:59:26] RECOVERY Disk Space is now: OK on redis1 i-000002b6 output: DISK OK [03:01:26] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [03:01:26] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [03:01:27] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [03:02:36] RECOVERY Current Load is now: OK on zeromq1 i-000002b7 output: OK - load average: 0.28, 0.15, 0.09 [03:03:06] RECOVERY Current Users is now: OK on zeromq1 i-000002b7 output: USERS OK - 0 users currently logged in [03:03:46] RECOVERY Disk Space is now: OK on zeromq1 i-000002b7 output: DISK OK [03:04:26] RECOVERY Free ram is now: OK on zeromq1 i-000002b7 output: OK: 89% free memory [03:05:49] RECOVERY Total Processes is now: OK on zeromq1 i-000002b7 output: PROCS OK: 80 processes [03:06:13] RECOVERY dpkg-check is now: OK on zeromq1 i-000002b7 output: All packages OK [03:07:13] PROBLEM Puppet freshness is now: CRITICAL on cn-wiki-db-lucid i-00000241 output: Puppet has not run in last 20 hours [03:14:13] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-1 i-0000020c output: Puppet has not run in last 20 hours [03:15:13] PROBLEM Puppet freshness is now: CRITICAL on demo-mysql1 i-00000256 output: Puppet has not run in last 20 hours [03:17:13] PROBLEM Puppet freshness is now: CRITICAL on dumps-1 i-00000170 output: Puppet has not run in last 20 hours [03:17:13] PROBLEM Puppet freshness is now: CRITICAL on wikistream-1 i-0000016e output: Puppet has not run in last 20 hours [03:18:13] PROBLEM Puppet freshness is now: CRITICAL on swift-be3 i-000001c9 output: Puppet has not run in last 20 hours [03:22:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-upload i-00000263 output: Puppet has not run in last 20 hours [03:22:13] PROBLEM Puppet freshness is now: CRITICAL on publicdata-administration i-0000019e output: Puppet has not run in last 20 hours [03:22:13] PROBLEM Puppet freshness is now: CRITICAL on resourceloader2-apache i-000001d7 output: Puppet has not run in last 20 hours [03:22:13] PROBLEM Puppet freshness is now: CRITICAL on test2 i-0000013c output: Puppet has not run in last 20 hours [03:22:13] PROBLEM Puppet freshness is now: CRITICAL on wikidata-dev-3 i-00000225 output: Puppet has not run in last 20 hours [03:23:13] PROBLEM Puppet freshness is now: CRITICAL on venus i-000000ea output: Puppet has not run in last 20 hours [03:23:13] PROBLEM Puppet freshness is now: CRITICAL on wmde-test i-000002ad output: Puppet has not run in last 20 hours [03:24:13] PROBLEM Puppet freshness is now: CRITICAL on wikisource-web i-000000fe output: Puppet has not run in last 20 hours [03:26:33] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [03:29:18] PROBLEM Puppet freshness is now: CRITICAL on bots-sql1 i-000000b5 output: Puppet has not run in last 20 hours [03:29:18] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [03:30:13] PROBLEM Puppet freshness is now: CRITICAL on bastion1 i-000000ba output: Puppet has not run in last 20 hours [03:30:13] PROBLEM Puppet freshness is now: CRITICAL on demo-web1 i-00000255 output: Puppet has not run in last 20 hours [03:30:13] PROBLEM Puppet freshness is now: CRITICAL on kripke i-00000268 output: Puppet has not run in last 20 hours [03:30:13] PROBLEM Puppet freshness is now: CRITICAL on labs-nfs1 i-0000005d output: Puppet has not run in last 20 hours [03:31:33] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [03:31:33] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [03:31:33] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [03:33:13] PROBLEM Puppet freshness is now: CRITICAL on bots-4 i-000000e8 output: Puppet has not run in last 20 hours [03:34:13] PROBLEM Puppet freshness is now: CRITICAL on labs-relay i-00000103 output: Puppet has not run in last 20 hours [03:35:13] PROBLEM Puppet freshness is now: CRITICAL on gerrit i-000000ff output: Puppet has not run in last 20 hours [03:35:13] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 17% free memory [03:36:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-backup i-000000f8 output: Puppet has not run in last 20 hours [03:37:13] PROBLEM Puppet freshness is now: CRITICAL on bots-2 i-0000009c output: Puppet has not run in last 20 hours [03:37:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache22 i-0000026f output: Puppet has not run in last 20 hours [03:37:13] PROBLEM Puppet freshness is now: CRITICAL on grail i-0000021e output: Puppet has not run in last 20 hours [03:37:13] PROBLEM Puppet freshness is now: CRITICAL on jenkins2 i-00000102 output: Puppet has not run in last 20 hours [03:37:13] PROBLEM Puppet freshness is now: CRITICAL on swift-be2 i-000001c8 output: Puppet has not run in last 20 hours [03:38:13] PROBLEM Puppet freshness is now: CRITICAL on nginx-dev1 i-000000f0 output: Puppet has not run in last 20 hours [03:39:14] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache23 i-00000270 output: Puppet has not run in last 20 hours [03:39:14] PROBLEM Puppet freshness is now: CRITICAL on memcache-puppet i-00000153 output: Puppet has not run in last 20 hours [03:40:13] PROBLEM Puppet freshness is now: CRITICAL on php5builds i-00000192 output: Puppet has not run in last 20 hours [03:42:13] PROBLEM Puppet freshness is now: CRITICAL on mobile-feeds i-000000c1 output: Puppet has not run in last 20 hours [03:43:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-dbdump i-000000d2 output: Puppet has not run in last 20 hours [03:43:13] PROBLEM Puppet freshness is now: CRITICAL on robh2 i-000001a2 output: Puppet has not run in last 20 hours [03:43:13] PROBLEM Puppet freshness is now: CRITICAL on swift-be4 i-000001ca output: Puppet has not run in last 20 hours [03:45:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-thumbproxy i-0000026b output: Puppet has not run in last 20 hours [03:45:13] PROBLEM Puppet freshness is now: CRITICAL on fundraising-civicrm i-00000169 output: Puppet has not run in last 20 hours [03:47:13] PROBLEM Puppet freshness is now: CRITICAL on wep i-000000c2 output: Puppet has not run in last 20 hours [03:47:13] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 14% free memory [03:49:18] PROBLEM Puppet freshness is now: CRITICAL on deployment-feed i-00000118 output: Puppet has not run in last 20 hours [03:51:13] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 15% free memory [03:53:13] PROBLEM Puppet freshness is now: CRITICAL on ee-prototype i-0000013d output: Puppet has not run in last 20 hours [03:55:13] PROBLEM Puppet freshness is now: CRITICAL on hugglewiki i-000000aa output: Puppet has not run in last 20 hours [03:55:13] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [03:56:33] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [03:57:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-transcoding i-00000105 output: Puppet has not run in last 20 hours [04:00:13] PROBLEM Puppet freshness is now: CRITICAL on bots-apache1 i-000000b0 output: Puppet has not run in last 20 hours [04:00:13] PROBLEM Puppet freshness is now: CRITICAL on secondinstance i-0000015b output: Puppet has not run in last 20 hours [04:00:13] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 96% free memory [04:01:34] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [04:01:34] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [04:01:34] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [04:02:13] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [04:03:28] PROBLEM Puppet freshness is now: CRITICAL on ganglia-collector i-000000b7 output: Puppet has not run in last 20 hours [04:05:39] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 17% free memory [04:07:19] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:11:19] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 3% free memory [04:12:09] PROBLEM Puppet freshness is now: CRITICAL on shop-analytics-main i-000001e6 output: Puppet has not run in last 20 hours [04:16:19] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:26:09] PROBLEM Puppet freshness is now: CRITICAL on building i-0000014d output: Puppet has not run in last 20 hours [04:26:39] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [04:30:39] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 3% free memory [04:31:09] PROBLEM Puppet freshness is now: CRITICAL on swift-fe1 i-000001d2 output: Puppet has not run in last 20 hours [04:31:39] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [04:31:39] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [04:31:39] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [04:35:39] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 97% free memory [04:41:09] PROBLEM Puppet freshness is now: CRITICAL on firstinstance i-0000013e output: Puppet has not run in last 20 hours [04:41:09] PROBLEM Puppet freshness is now: CRITICAL on log1 i-00000239 output: Puppet has not run in last 20 hours [04:44:09] PROBLEM Puppet freshness is now: CRITICAL on pybal-precise i-00000289 output: Puppet has not run in last 20 hours [04:46:09] PROBLEM Puppet freshness is now: CRITICAL on reportcard2 i-000001ea output: Puppet has not run in last 20 hours [04:56:39] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [05:01:09] PROBLEM Puppet freshness is now: CRITICAL on deployment-imagescaler01 i-0000025a output: Puppet has not run in last 20 hours [05:01:09] PROBLEM Puppet freshness is now: CRITICAL on worker1 i-00000208 output: Puppet has not run in last 20 hours [05:01:39] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [05:01:39] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [05:01:39] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [05:09:09] PROBLEM Puppet freshness is now: CRITICAL on migration1 i-00000261 output: Puppet has not run in last 20 hours [05:10:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-cache-bits i-00000264 output: Puppet has not run in last 20 hours [05:13:45] PROBLEM Current Load is now: CRITICAL on dumps-incr i-000002bb output: Connection refused by host [05:14:25] PROBLEM Current Users is now: CRITICAL on dumps-incr i-000002bb output: Connection refused by host [05:15:05] PROBLEM Disk Space is now: CRITICAL on dumps-incr i-000002bb output: Connection refused by host [05:15:45] PROBLEM Free ram is now: CRITICAL on dumps-incr i-000002bb output: Connection refused by host [05:16:55] PROBLEM Total Processes is now: CRITICAL on dumps-incr i-000002bb output: Connection refused by host [05:17:15] PROBLEM Puppet freshness is now: CRITICAL on catsort-pub i-000001cc output: Puppet has not run in last 20 hours [05:17:35] PROBLEM dpkg-check is now: CRITICAL on dumps-incr i-000002bb output: Connection refused by host [05:19:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache21 i-0000026d output: Puppet has not run in last 20 hours [05:19:16] PROBLEM Puppet freshness is now: CRITICAL on labs-build1 i-0000006b output: Puppet has not run in last 20 hours [05:21:15] PROBLEM Puppet freshness is now: CRITICAL on bots-sql2 i-000000af output: Puppet has not run in last 20 hours [05:21:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-apache20 i-0000026c output: Puppet has not run in last 20 hours [05:21:15] PROBLEM Puppet freshness is now: CRITICAL on dumps-2 i-00000257 output: Puppet has not run in last 20 hours [05:23:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-sql i-000000d0 output: Puppet has not run in last 20 hours [05:23:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-wmsearch i-000000e1 output: Puppet has not run in last 20 hours [05:23:15] PROBLEM Puppet freshness is now: CRITICAL on dumps-3 i-0000017c output: Puppet has not run in last 20 hours [05:23:15] PROBLEM Puppet freshness is now: CRITICAL on hugglewa-1 i-000001e0 output: Puppet has not run in last 20 hours [05:23:15] PROBLEM Puppet freshness is now: CRITICAL on master i-0000007a output: Puppet has not run in last 20 hours [05:23:16] PROBLEM Puppet freshness is now: CRITICAL on mingledbtest i-00000283 output: Puppet has not run in last 20 hours [05:25:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [05:25:15] PROBLEM Puppet freshness is now: CRITICAL on p-b i-000000ae output: Puppet has not run in last 20 hours [05:26:15] PROBLEM Puppet freshness is now: CRITICAL on feeds i-000000fa output: Puppet has not run in last 20 hours [05:26:15] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot0 i-00000296 output: Puppet has not run in last 20 hours [05:26:45] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [05:27:15] PROBLEM Puppet freshness is now: CRITICAL on asher1 i-0000003a output: Puppet has not run in last 20 hours [05:27:15] PROBLEM Puppet freshness is now: CRITICAL on pediapress-ocg2 i-00000234 output: Puppet has not run in last 20 hours [05:27:15] PROBLEM Puppet freshness is now: CRITICAL on udp-filter i-000001df output: Puppet has not run in last 20 hours [05:28:15] PROBLEM Puppet freshness is now: CRITICAL on bob i-0000012d output: Puppet has not run in last 20 hours [05:28:15] PROBLEM Puppet freshness is now: CRITICAL on bots-dev i-00000190 output: Puppet has not run in last 20 hours [05:28:15] PROBLEM Puppet freshness is now: CRITICAL on e3 i-00000291 output: Puppet has not run in last 20 hours [05:28:15] PROBLEM Puppet freshness is now: CRITICAL on testing-groupchange i-00000205 output: Puppet has not run in last 20 hours [05:29:15] PROBLEM Puppet freshness is now: CRITICAL on bots-labs i-0000015e output: Puppet has not run in last 20 hours [05:29:15] PROBLEM Puppet freshness is now: CRITICAL on demo-deployment1 i-00000276 output: Puppet has not run in last 20 hours [05:29:15] PROBLEM Puppet freshness is now: CRITICAL on fundraising-db i-0000015c output: Puppet has not run in last 20 hours [05:29:15] PROBLEM Puppet freshness is now: CRITICAL on otrs-jgreen i-0000015a output: Puppet has not run in last 20 hours [05:29:15] PROBLEM Puppet freshness is now: CRITICAL on ve-nodejs i-00000245 output: Puppet has not run in last 20 hours [05:30:15] PROBLEM Puppet freshness is now: CRITICAL on dumps-4 i-0000017d output: Puppet has not run in last 20 hours [05:30:15] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test4 i-000002a2 output: Puppet has not run in last 20 hours [05:30:15] PROBLEM Puppet freshness is now: CRITICAL on incubator-common i-00000254 output: Puppet has not run in last 20 hours [05:30:15] PROBLEM Puppet freshness is now: CRITICAL on varnish i-000001ac output: Puppet has not run in last 20 hours [05:31:15] PROBLEM Puppet freshness is now: CRITICAL on tw-next i-0000027e output: Puppet has not run in last 20 hours [05:32:15] PROBLEM Puppet freshness is now: CRITICAL on bots-nfs i-000000b1 output: Puppet has not run in last 20 hours [05:32:15] PROBLEM Puppet freshness is now: CRITICAL on outreacheval i-0000012e output: Puppet has not run in last 20 hours [05:32:15] PROBLEM Puppet freshness is now: CRITICAL on webserver-lcarr i-00000134 output: Puppet has not run in last 20 hours [05:32:35] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [05:32:35] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [05:32:35] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [05:34:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-jobrunner05 i-0000028c output: Puppet has not run in last 20 hours [05:36:15] PROBLEM Puppet freshness is now: CRITICAL on bots-sql3 i-000000b4 output: Puppet has not run in last 20 hours [05:36:15] PROBLEM Puppet freshness is now: CRITICAL on build-precise1 i-00000273 output: Puppet has not run in last 20 hours [05:36:15] PROBLEM Puppet freshness is now: CRITICAL on en-wiki-db-lucid i-0000023b output: Puppet has not run in last 20 hours [05:36:15] PROBLEM Puppet freshness is now: CRITICAL on exim-test i-00000265 output: Puppet has not run in last 20 hours [05:36:15] PROBLEM Puppet freshness is now: CRITICAL on fr-wiki-db-precise i-0000023e output: Puppet has not run in last 20 hours [05:36:16] PROBLEM Puppet freshness is now: CRITICAL on incubator-apache i-00000211 output: Puppet has not run in last 20 hours [05:36:16] PROBLEM Puppet freshness is now: CRITICAL on labs-realserver i-00000104 output: Puppet has not run in last 20 hours [05:36:17] PROBLEM Puppet freshness is now: CRITICAL on nginx-ffuqua-doom1-3 i-00000196 output: Puppet has not run in last 20 hours [05:36:17] PROBLEM Puppet freshness is now: CRITICAL on pediapress-ocg1 i-00000233 output: Puppet has not run in last 20 hours [05:36:18] PROBLEM Puppet freshness is now: CRITICAL on pediapress-packager i-000001e4 output: Puppet has not run in last 20 hours [05:36:18] PROBLEM Puppet freshness is now: CRITICAL on scribunto i-0000022c output: Puppet has not run in last 20 hours [05:36:19] PROBLEM Puppet freshness is now: CRITICAL on simplewikt i-00000149 output: Puppet has not run in last 20 hours [05:37:15] PROBLEM Puppet freshness is now: CRITICAL on pageviews i-000000b2 output: Puppet has not run in last 20 hours [05:38:15] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test3 i-0000025b output: Puppet has not run in last 20 hours [05:38:15] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot1 i-00000251 output: Puppet has not run in last 20 hours [05:41:15] PROBLEM Puppet freshness is now: CRITICAL on swift-aux2 i-0000024c output: Puppet has not run in last 20 hours [05:42:15] PROBLEM Puppet freshness is now: CRITICAL on bastion-restricted1 i-0000019b output: Puppet has not run in last 20 hours [05:42:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-mc i-0000021b output: Puppet has not run in last 20 hours [05:42:15] PROBLEM Puppet freshness is now: CRITICAL on nova-dev3 i-000000e9 output: Puppet has not run in last 20 hours [05:42:15] PROBLEM Puppet freshness is now: CRITICAL on patchtest i-000000f1 output: Puppet has not run in last 20 hours [05:42:15] PROBLEM Puppet freshness is now: CRITICAL on vumi i-000001e5 output: Puppet has not run in last 20 hours [05:43:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-squid i-000000dc output: Puppet has not run in last 20 hours [05:43:15] PROBLEM Puppet freshness is now: CRITICAL on embed-sandbox i-000000d1 output: Puppet has not run in last 20 hours [05:43:16] PROBLEM Puppet freshness is now: CRITICAL on opengrok-web i-000001e1 output: Puppet has not run in last 20 hours [05:43:16] PROBLEM Puppet freshness is now: CRITICAL on swift-aux1 i-0000024b output: Puppet has not run in last 20 hours [05:43:16] PROBLEM Puppet freshness is now: CRITICAL on vivek-puppet i-000000ca output: Puppet has not run in last 20 hours [05:45:15] PROBLEM Puppet freshness is now: CRITICAL on dev-solr i-00000152 output: Puppet has not run in last 20 hours [05:46:15] PROBLEM Puppet freshness is now: CRITICAL on demo-web2 i-00000285 output: Puppet has not run in last 20 hours [05:46:15] PROBLEM Puppet freshness is now: CRITICAL on en-wiki-db-precise i-0000023c output: Puppet has not run in last 20 hours [05:46:15] PROBLEM Puppet freshness is now: CRITICAL on ubuntu1-pgehres i-000000fb output: Puppet has not run in last 20 hours [05:47:15] PROBLEM Puppet freshness is now: CRITICAL on maps-test2 i-00000253 output: Puppet has not run in last 20 hours [05:48:15] PROBLEM Puppet freshness is now: CRITICAL on bots-1 i-000000a9 output: Puppet has not run in last 20 hours [05:50:15] PROBLEM Puppet freshness is now: CRITICAL on build1 i-000002b3 output: Puppet has not run in last 20 hours [05:50:15] PROBLEM Puppet freshness is now: CRITICAL on deployment-nfs-memc i-000000d7 output: Puppet has not run in last 20 hours [05:51:15] PROBLEM Puppet freshness is now: CRITICAL on hugglewa-db i-00000188 output: Puppet has not run in last 20 hours [05:51:15] PROBLEM Puppet freshness is now: CRITICAL on maps-test3 i-0000028f output: Puppet has not run in last 20 hours [05:51:15] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap2 i-00000238 output: Puppet has not run in last 20 hours [05:52:15] PROBLEM Puppet freshness is now: CRITICAL on translation-memory-1 i-0000013a output: Puppet has not run in last 20 hours [05:56:45] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [06:02:45] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [06:02:45] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [06:02:45] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [06:27:26] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [06:28:05] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 13% free memory [06:33:26] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [06:33:26] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [06:33:26] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [06:58:10] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [07:03:10] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [07:03:10] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [07:03:10] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [07:28:10] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [07:33:10] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [07:33:10] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [07:33:10] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [07:38:10] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [07:42:10] PROBLEM Puppet freshness is now: CRITICAL on blamemaps-m1small i-000002a1 output: Puppet has not run in last 20 hours [07:54:57] hey [07:56:16] hello :) [07:58:10] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [08:04:24] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [08:04:24] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [08:04:24] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [08:09:14] PROBLEM Puppet freshness is now: CRITICAL on mobile-testing i-00000271 output: Puppet has not run in last 20 hours [08:11:14] PROBLEM Puppet freshness is now: CRITICAL on precise-test i-00000231 output: Puppet has not run in last 20 hours [08:20:14] PROBLEM Puppet freshness is now: CRITICAL on bots-3 i-000000e5 output: Puppet has not run in last 20 hours [08:22:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test5 i-000002a7 output: Puppet has not run in last 20 hours [08:22:14] PROBLEM Puppet freshness is now: CRITICAL on mwreview i-000002ae output: Puppet has not run in last 20 hours [08:23:14] PROBLEM Puppet freshness is now: CRITICAL on incubator-bot2 i-00000252 output: Puppet has not run in last 20 hours [08:23:41] is there hashar somewhere in venue [08:24:04] 06/02/2012 - 08:24:04 - Updating keys for wikinaut at /export/home/etherpad/wikinaut [08:24:14] PROBLEM Puppet freshness is now: CRITICAL on ganglia-test6 i-000002af output: Puppet has not run in last 20 hours [08:24:15] 06/02/2012 - 08:24:14 - Updating keys for wikinaut at /export/home/bastion/wikinaut [08:25:14] PROBLEM Puppet freshness is now: CRITICAL on localpuppet1 i-0000020b output: Puppet has not run in last 20 hours [08:26:44] RECOVERY Free ram is now: OK on dumps-incr i-000002bb output: OK: 91% free memory [08:27:44] RECOVERY Total Processes is now: OK on dumps-incr i-000002bb output: PROCS OK: 96 processes [08:27:49] RECOVERY dpkg-check is now: OK on dumps-incr i-000002bb output: All packages OK [08:28:44] RECOVERY Current Load is now: OK on dumps-incr i-000002bb output: OK - load average: 0.27, 0.50, 0.49 [08:29:14] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [08:29:24] RECOVERY Current Users is now: OK on dumps-incr i-000002bb output: USERS OK - 2 users currently logged in [08:30:14] 06/02/2012 - 08:30:14 - Updating keys for multichill at /export/home/bastion/multichill [08:30:44] RECOVERY Disk Space is now: OK on dumps-incr i-000002bb output: DISK OK [08:30:58] good mornin [08:30:59] g [08:31:23] hey [08:31:27] was looking for you [08:31:32] hashar: where are you [08:31:47] asleep ;-D [08:31:48] still [08:32:04] petan|hackaton: is the seat on your left free? [08:32:06] hi hashar [08:32:12] sumanah: ;-) [08:32:14] PROBLEM Puppet freshness is now: CRITICAL on maps-tilemill1 i-00000294 output: Puppet has not run in last 20 hours [08:32:14] hashar: I just, that it seems that we have ipv6 on labs now, so maybe we could make some progress on ipv6 on beta [08:32:33] petan|hackaton: oh really ? ??? [08:32:39] was it enabled by Ryan like yesterday? [08:32:50] hashar: I don't know, Danny_B wanted something as well, but right now the seat is free [08:33:03] comin [08:33:17] http://ipv6test.wmflabs.org/wiki/index.php?title=Special:UserLogin&returnto=Main+Page check this [08:33:58] hashar: take it, it's free ;-) [08:34:24] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [08:34:24] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [08:34:24] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [08:46:02] mutante: hi [08:48:14] 06/02/2012 - 08:48:13 - Creating a home directory for yuvipanda at /export/home/mobile/yuvipanda [08:49:13] 06/02/2012 - 08:49:12 - Updating keys for yuvipanda at /export/home/mobile/yuvipanda [08:49:14] PROBLEM Puppet freshness is now: CRITICAL on mailman-01 i-00000235 output: Puppet has not run in last 20 hours [08:59:14] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [09:03:45] PROBLEM Current Load is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:04:25] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [09:04:25] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [09:04:25] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [09:04:25] PROBLEM Current Users is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:05:05] PROBLEM Disk Space is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:05:45] PROBLEM Free ram is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:06:55] PROBLEM Total Processes is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:07:35] PROBLEM dpkg-check is now: CRITICAL on mobile-wlm i-000002bc output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:12:13] 06/02/2012 - 09:12:13 - Creating a home directory for multichill at /export/home/mobile/multichill [09:13:11] 06/02/2012 - 09:13:11 - Updating keys for multichill at /export/home/mobile/multichill [09:16:23] !log deployment-prep Rebooting -dbdump it could not mount some NFS export and waiting for user input. [09:16:25] Logged the message, Master [09:17:45] RECOVERY Current Load is now: OK on deployment-dbdump i-000000d2 output: OK - load average: 2.18, 0.82, 0.29 [09:17:45] RECOVERY Disk Space is now: OK on deployment-dbdump i-000000d2 output: DISK OK [09:17:45] RECOVERY Current Users is now: OK on deployment-dbdump i-000000d2 output: USERS OK - 0 users currently logged in [09:18:25] RECOVERY Free ram is now: OK on deployment-dbdump i-000000d2 output: OK: 94% free memory [09:19:05] RECOVERY SSH is now: OK on deployment-dbdump i-000000d2 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [09:23:44] PROBLEM Current Load is now: CRITICAL on deployment-bastion i-000002bd output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:24:24] PROBLEM Current Users is now: CRITICAL on deployment-bastion i-000002bd output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:25:04] PROBLEM Disk Space is now: CRITICAL on deployment-bastion i-000002bd output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:25:44] PROBLEM Free ram is now: CRITICAL on deployment-bastion i-000002bd output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:26:44] RECOVERY Total Processes is now: OK on deployment-dbdump i-000000d2 output: PROCS OK: 91 processes [09:26:49] RECOVERY dpkg-check is now: OK on deployment-dbdump i-000000d2 output: All packages OK [09:26:54] PROBLEM Total Processes is now: CRITICAL on deployment-bastion i-000002bd output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:27:34] PROBLEM dpkg-check is now: CRITICAL on deployment-bastion i-000002bd output: CHECK_NRPE: Error - Could not complete SSL handshake. [09:29:14] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [09:29:57] any cloudadmin around who can give jorm access to bastion? [09:34:27] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [09:34:27] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [09:34:27] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [09:43:55] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 78 MB (5% inode=58%): [09:59:25] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [10:04:55] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [10:04:55] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [10:05:35] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [10:19:15] PROBLEM Puppet freshness is now: CRITICAL on bots-cb i-0000009e output: Puppet has not run in last 20 hours [10:20:20] !log deployment-prep root: setting up bastion [10:20:21] Logged the message, Master [10:29:25] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [10:32:54] hashar: Error 400 on SERVER: Could not find class labs::beta::bastion [10:34:55] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [10:34:55] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [10:35:05] RECOVERY Disk Space is now: OK on deployment-bastion i-000002bd output: DISK OK [10:35:35] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [10:35:35] RECOVERY dpkg-check is now: OK on deployment-bastion i-000002bd output: All packages OK [10:35:35] RECOVERY Free ram is now: OK on deployment-bastion i-000002bd output: OK: 93% free memory [10:36:55] RECOVERY Total Processes is now: OK on deployment-bastion i-000002bd output: PROCS OK: 108 processes [10:38:30] mutante: there is problem with puppet again [10:38:45] RECOVERY Current Load is now: OK on deployment-bastion i-000002bd output: OK - load average: 0.03, 0.32, 0.32 [10:38:57] mutante: doesn't send a check ping to nagios [10:39:25] RECOVERY Current Users is now: OK on deployment-bastion i-000002bd output: USERS OK - 1 users currently logged in [10:40:02] petan|hackaton: did you get my poke yesterday? [10:40:05] about email on labs? [10:40:26] it seems to work for other instances - ipv6 for example [10:40:28] projects* [10:41:55] Thehelpfulone: I will talk to Ryan as soon as he arrive here [10:59:54] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [11:03:35] PROBLEM dpkg-check is now: CRITICAL on deployment-bastion i-000002bd output: DPKG CRITICAL dpkg reports broken packages [11:04:55] petan|hackaton: ^ ok thanks [11:04:55] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [11:04:55] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [11:05:35] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [11:06:17] 06/02/2012 - 11:06:17 - Creating a home directory for anomie at /export/home/bastion/anomie [11:07:17] 06/02/2012 - 11:07:17 - Updating keys for anomie at /export/home/bastion/anomie [11:07:47] 06/02/2012 - 11:07:47 - Creating a home directory for anomie at /export/home/tutorial/anomie [11:08:46] 06/02/2012 - 11:08:46 - Updating keys for anomie at /export/home/tutorial/anomie [11:13:35] RECOVERY dpkg-check is now: OK on deployment-bastion i-000002bd output: All packages OK [11:30:05] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [11:35:05] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [11:35:05] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [11:35:35] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [11:52:16] 06/02/2012 - 11:52:15 - Creating a home directory for bharris at /export/home/bastion/bharris [11:53:15] PROBLEM Puppet freshness is now: CRITICAL on ipv6test1 i-00000282 output: Puppet has not run in last 20 hours [11:53:17] 06/02/2012 - 11:53:16 - Updating keys for bharris at /export/home/bastion/bharris [11:55:15] PROBLEM Puppet freshness is now: CRITICAL on rds i-00000207 output: Puppet has not run in last 20 hours [11:55:15] PROBLEM Puppet freshness is now: CRITICAL on upload-wizard i-0000021c output: Puppet has not run in last 20 hours [11:58:55] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [12:00:05] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [12:00:15] 06/02/2012 - 12:00:15 - Creating a home directory for kolossos at /export/home/bastion/kolossos [12:01:17] 06/02/2012 - 12:01:16 - Updating keys for kolossos at /export/home/bastion/kolossos [12:05:05] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [12:05:05] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [12:05:35] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [12:06:55] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 78 MB (5% inode=58%): [12:19:17] 06/02/2012 - 12:19:16 - Creating a home directory for dartar at /export/home/bastion/dartar [12:20:17] 06/02/2012 - 12:20:16 - Updating keys for dartar at /export/home/bastion/dartar [12:27:51] hi, could you please check a rootcause of a problem, which I experience, basically I can't get access to the created account `vz' on Labs. [12:28:20] thank you in advance [12:29:18] vzapolskiy: via the web interface? [12:29:22] or via ssh? [12:29:26] via ssh [12:29:31] to bastion? [12:29:35] yes [12:30:05] PROBLEM host: ganglia-test2 is DOWN address: i-00000250 CRITICAL - Host Unreachable (i-00000250) [12:32:06] Ryan_Lane: the semantic search is broken: https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/mainlabel%3D-2D [12:32:09] shows no projects [12:32:14] I know :( [12:32:17] that's the link from Number of projects: [12:32:26] I think petan found a link - https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&q=[[Resource+Type%3A%3Aproject]]&p=format%3Dbroadtable%2Fheaders%3Dshow%2Fmainlabel%3D-2D%2Flink%3Dall%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3F%0A%3FMember%0A%3FDescription%0A&limit=500&eq=no [12:34:13] Ryan_Lane, any idea why jorm's sudo password on the mwreview instance might not work? the sudoer policy is set for ALL according to https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaSudoer&action=modify&sudoername=default&project=mwreview [12:34:37] mine is working fine, and he's trying with his labsconsole pw to no avail [12:35:05] PROBLEM host: aggregator-test is DOWN address: i-0000024d CRITICAL - Host Unreachable (i-0000024d) [12:35:05] PROBLEM host: aggregator-test3 is DOWN address: i-00000293 CRITICAL - Host Unreachable (i-00000293) [12:35:08] lemme see [12:35:35] PROBLEM host: aggregator1 is DOWN address: i-0000010c CRITICAL - Host Unreachable (i-0000010c) [12:36:35] should work... [12:37:26] Eloquence: whilst you're here, can you install shorturl on the ipv6 testwiki? or is it going to be around only for a short while? [12:37:51] that's pretty throwaway, why do you want it? [12:38:36] ah if it's a throw away then no problem, otherwise we all <3 pretty urls [12:38:53] jorm: can you try sudoing for me [12:38:54] ? [12:39:03] heh [12:39:09] not sure I'd call the shorturl urls pretty ;) [12:39:37] three incorrect attempts. [12:39:48]