[00:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [00:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [00:24:04] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [00:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [00:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [00:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [00:50:44] RECOVERY dpkg-check is now: OK on driver-dev driver-dev output: All packages OK [00:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [00:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [00:54:04] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [00:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [00:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [00:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [01:19:44] PROBLEM dpkg-check is now: CRITICAL on driver-dev driver-dev output: DPKG CRITICAL dpkg reports broken packages [01:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [01:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [01:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [01:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [01:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [01:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [01:34:44] RECOVERY dpkg-check is now: OK on driver-dev driver-dev output: All packages OK [01:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [01:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [01:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [01:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [01:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [01:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [02:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [02:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [02:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [02:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [02:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [02:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [02:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [02:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [02:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [02:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [02:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [02:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [03:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [03:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [03:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [03:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [03:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [03:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [03:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [03:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [03:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [03:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [03:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [03:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [04:20:44] PROBLEM dpkg-check is now: CRITICAL on driver-dev driver-dev output: DPKG CRITICAL dpkg reports broken packages [04:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [04:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [04:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [04:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [04:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [04:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [04:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [04:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [04:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [04:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [04:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [04:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [05:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [05:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [05:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [05:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [05:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [05:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [05:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [05:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [05:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [05:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [05:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [05:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [06:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [06:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [06:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [06:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [06:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [06:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [06:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [06:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [06:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [06:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [06:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [06:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [07:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [07:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [07:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [07:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [07:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [07:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [07:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [07:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [07:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [07:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [07:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [07:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [08:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [08:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [08:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [08:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [08:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [08:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [08:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [08:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [08:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [08:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [08:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [08:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [09:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [09:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [09:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [09:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [09:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [09:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [09:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [09:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [09:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [09:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [09:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [09:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [10:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [10:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [10:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [10:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [10:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [10:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [10:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [10:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [10:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [10:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [10:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [10:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [11:09:44] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 17% free memory [11:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [11:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [11:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [11:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [11:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [11:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [11:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [11:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [11:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [11:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [11:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [11:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [12:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [12:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [12:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [12:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [12:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [12:25:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [12:41:14] PROBLEM Free ram is now: WARNING on deployment-backup deployment-backup output: Warning: 18% free memory [12:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [12:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [12:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [12:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [12:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [12:54:44] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 26% free memory [12:55:44] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [13:21:14] RECOVERY Free ram is now: OK on deployment-backup deployment-backup output: OK: 20% free memory [13:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [13:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [13:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [13:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [13:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [13:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [13:36:45] anyone got an idea what gadget could cause TMH to fail on http://commons.wikimedia.beta.wmflabs.org/wiki/File:Sheep.ogv? it works if loaded in debug mode http://commons.wikimedia.beta.wmflabs.org/wiki/File:Sheep.ogv?debug=true [13:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [13:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [13:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [13:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [13:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [13:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [14:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [14:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [14:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [14:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [14:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [14:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [14:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [14:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [14:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [14:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [14:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [14:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [15:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [15:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [15:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [15:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [15:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [15:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [15:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [15:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [15:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [15:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [15:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [15:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [16:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [16:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [16:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [16:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [16:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [16:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [16:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [16:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [16:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [16:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [16:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [16:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [17:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [17:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [17:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [17:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [17:24:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [17:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [17:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [17:53:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [17:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [17:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [17:54:24] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [17:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [18:02:01] petan: is it still going on ? [18:02:04] the borked in labs ? [18:21:09] * Damianz sends LeslieCarr to fix it [18:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [18:23:54] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [18:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [18:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [18:24:34] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [18:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [18:34:24] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 19% free memory [18:36:20] :o [18:48:54] PROBLEM Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 19% free memory [18:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [18:54:04] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [18:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [18:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [18:54:34] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [18:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [19:11:16] 02/03/2012 - 19:11:16 - Creating a home directory for declerambaul at /export/home/mobile-stats/declerambaul [19:12:15] 02/03/2012 - 19:12:15 - Updating keys for declerambaul [19:20:15] 02/03/2012 - 19:20:15 - Updating keys for declerambaul [19:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [19:24:04] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [19:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [19:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [19:24:34] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [19:24:53] who can help declerambaul with setting up his labs access? he can't get in to bastion even though he has registerd his public key [19:25:44] RECOVERY dpkg-check is now: OK on driver-dev driver-dev output: All packages OK [19:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [19:48:44] PROBLEM dpkg-check is now: CRITICAL on driver-dev driver-dev output: DPKG CRITICAL dpkg reports broken packages [19:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [19:54:04] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [19:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [19:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [19:54:44] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [19:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [20:23:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [20:24:04] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [20:24:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [20:24:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [20:24:44] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [20:25:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [20:38:44] RECOVERY dpkg-check is now: OK on driver-dev driver-dev output: All packages OK [20:50:17] drdee: did declerambaul get help? [20:50:26] or diederik [20:50:36] not yet :) [20:51:27] declerambaul: well tell me about your current state? [20:51:44] thanks for helping me out. [20:51:52] no not yet :) [20:52:14] i am trying to connect to bastion, but there seems to be a problem with my ssh setup [20:52:47] i added my public key to https://labsconsole.wikimedia.org/wiki/Special:NovaKey [20:52:48] tell me more [20:52:51] OS? [20:52:55] mac [20:53:10] wiki user name? [20:53:16] declerambaul [20:53:26] when did you add the key? [20:53:54] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [20:54:04] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [20:54:06] a couple of days ago i added it. then i deleted it and re-added it today [20:54:14] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [20:54:14] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [20:54:36] labs-home-wm: 02/03/2012 - 19:20:15 - Updating keys for declerambaul [20:54:44] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [20:55:00] ok, well obvious problem is that you're a member of a project but not also a member of bastion [20:55:20] are you WMF? [20:55:24] contractor [20:55:44] fkaelin@wikimedia.org [20:55:54] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [20:56:48] so, you need someone on this list: https://labsconsole.wikimedia.org/wiki/Special:ListUsers/sysop [20:56:52] to add you to bastion [20:58:08] is ryan around? [20:58:23] that somebody would be lane? [20:58:33] yes [20:58:43] notpeter: want to fix that quick? ^^ [20:58:55] while you wait for review ;) [20:59:28] what's going on? [21:00:02] hi peter, fabian is working with me (declerambaul) on the reportcard project [21:00:09] he is an external contractor [21:00:13] notpeter: declerambaul needs to be a member of bastion or he can't access the boxen for the project that he's already a member of [21:00:21] and he has labs account but cannot acccess bastion yet [21:02:07] okie dokie [21:06:27] ok, has been added in labsconsole [21:06:46] thanks! [21:07:12] 02/03/2012 - 21:07:11 - Creating a home directory for declerambaul at /export/home/bastion/declerambaul [21:07:38] thanks everybody! i am now one of you. [21:08:13] 02/03/2012 - 21:08:12 - Updating keys for declerambaul [21:08:36] declerambaul: works? [21:09:01] yep, logged into bastion. [21:09:12] and to the other project? [21:09:24] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 21% free memory [21:09:27] nope. doesn't work. [21:09:27] ssh analytics [21:09:37] Permission denied (publickey). [21:10:54] need agent forwarding [21:11:34] or other method ;) [21:11:44] !access [21:11:44] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [21:21:43] PROBLEM dpkg-check is now: CRITICAL on driver-dev driver-dev output: DPKG CRITICAL dpkg reports broken packages [21:24:13] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [21:24:23] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [21:24:23] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [21:24:48] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [21:24:53] PROBLEM Current Load is now: CRITICAL on bob bob output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:25:03] PROBLEM Current Users is now: CRITICAL on bob bob output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:25:03] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [21:25:48] PROBLEM Disk Space is now: CRITICAL on bob bob output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:26:23] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [21:26:33] PROBLEM Free ram is now: CRITICAL on bob bob output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:27:43] PROBLEM Total Processes is now: CRITICAL on bob bob output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:28:13] PROBLEM dpkg-check is now: CRITICAL on bob bob output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:31:43] RECOVERY dpkg-check is now: OK on driver-dev driver-dev output: All packages OK [21:39:02] jeremyb: i do 'ssh -A declerambaul@bastion.wmflabs.org' and then 'ssh analytics', but i still get a 'Permission denied (publickey).' [21:51:07] declerambaul: sorry, didn't see this until now [21:51:14] declerambaul: what happens when you 'ssh -A declerambaul@bastion.wmflabs.org' ? [21:51:22] you get a prompt of some kind? [21:51:46] yeah that works. [21:51:54] i mean before the shell [21:52:09] ssh -A declerambaul@bastion.wmflabs.org [21:52:10] does it give you a password or passphrase prompt? [21:52:10] Last login: Fri Feb 3 21:37:51 2012 from pool-173-52-12-54.nycmny.east.verizon.net [21:52:10] declerambaul@bastion1:~$ [21:52:15] nope [21:52:15] or say anything? [21:52:28] hrmm [21:54:13] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [21:54:23] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [21:54:33] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [21:55:03] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [21:55:03] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [21:56:23] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [21:57:25] declerambaul: drdee diederik: instance state for I-000000e2 aka analytics is pending according to labsconsole [21:57:31] can anyone log in? [21:57:57] can you check console output? [21:58:21] hmmmm. maybe i am trying to log onto the wrong instance. i just assumed the analytics team would have the analytics instance :) [21:58:44] no, it's the right project and it's the only instance in that project [21:59:06] see https://labsconsole.wikimedia.org/wiki/Nova_Resource:Mobile-stats [22:00:10] declerambaul: can you check console output? [22:01:33] * Damianz stretched out [22:07:58] jeremyb: sorry for the delay. [22:08:26] what do you mean by output? i posted everything that i see when i try to connect. [22:09:04] declerambaul: https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-000000e2 [22:09:09] click get console output [22:09:16] look for something interesting [22:09:21] report what you see ;) [22:09:40] talk to diederik and andre and if they also can't get in then reboot the box [22:09:50] No Nova credentials found for your account. [22:09:56] I am confused [22:09:58] log out of the wiki and back in [22:10:11] rinse, repeat [22:11:51] ups: [22:12:01] Jan 12 20:36:01 i-000000e2 nrpe[11077]: Allowing connections from: 10.4.0.34 [22:12:02] [665508.775312] Out of memory: kill process 17033 (bash) score 81538 or a child [22:12:16] [816480.550380] INFO: task apt-get:16572 blocked for more than 120 seconds. [22:12:17] [816480.558903] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [22:12:31] i hope you don't paste *too* much... [22:12:47] anyway, did you get ahold of them? [22:13:30] nope. but i will take it up with them. i was under the impression that people were working on it. [22:14:09] maybe have to reboot from the wiki [22:17:14] hi fabian [22:17:19] what's the problem? [22:17:39] drdee: can you login to analytics? [22:18:16] yes [22:19:41] diederik: any objection to rebooting? [22:19:45] no [22:19:52] but don't think that would solve it [22:20:06] 03 21:57:25 < jeremyb> declerambaul: drdee diederik: instance state for I-000000e2 aka analytics is pending according to labsconsole [22:20:15] idk why but maybe it means something [22:20:19] oh okay [22:20:24] go ahead [22:20:46] i can't reboot it [22:21:25] :D [22:21:33] RECOVERY Free ram is now: OK on bob bob output: OK: 90% free memory [22:21:48] bob bob? [22:22:39] bob in dutch means (b)ewust (o)nbeschonken (b)estuurder [22:22:43] RECOVERY Total Processes is now: OK on bob bob output: PROCS OK: 82 processes [22:22:48] which means: consiously not drinking while driving [22:22:54]