[00:21:41] nimish_g: did you get your new instance sorted? [00:22:26] chrismcmahon has some pretty bad internets [00:24:21] Anyone know how to update the code beta is running from? [00:24:35] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [00:24:40] vaguely [00:24:43] i need foods though [00:24:45] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [00:24:54] johnduhart knows better [00:24:55] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [00:24:58] petan is asleep [00:25:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [00:25:14] bbiab [00:25:15] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [00:25:19] I can't do anything. [00:25:19] I should probably know, but at half past midnight on a friday evening, i don't really care :p [00:25:34] Reedy: You have access? [00:25:44] I've no idea what I've got access to or not [00:26:01] Then you don't and you can't without root access iirc [00:26:05] let me look [00:26:45] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [00:28:48] the fuck [00:28:49] -rw-rwSr-- 1 mah depops 229 2012-01-10 16:06 thumb_handler.php [00:28:55] S. [00:29:00] Sticky? [00:29:24] Reedy: In any case you can't upate it without being in the project. [00:30:19] jeremyb: no, I haven't figured out how to give myself sudo [00:30:39] I'm a member of deployment-prep [00:32:17] PROBLEM dpkg-check is now: CRITICAL on driver-dev driver-dev output: DPKG CRITICAL dpkg reports broken packages [00:34:27] PROBLEM Current Load is now: CRITICAL on driver-dev2 driver-dev2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:35:07] PROBLEM Current Users is now: CRITICAL on driver-dev2 driver-dev2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:35:47] PROBLEM Disk Space is now: CRITICAL on driver-dev2 driver-dev2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:36:37] PROBLEM Free ram is now: CRITICAL on driver-dev2 driver-dev2 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:38:47] PROBLEM dpkg-check is now: CRITICAL on driver-dev2 driver-dev2 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:38:47] PROBLEM Total Processes is now: CRITICAL on driver-dev2 driver-dev2 output: Connection refused or timed out [00:41:22] Reedy: [00:41:24] Oh [00:41:28] Then in that case [00:41:53] on deployment-nfs-memc [00:42:51] cd /mnt/export/apache/common/live [00:42:53] svn up [00:44:56] Oh isn't this lovely [00:44:57] login as: reedy [00:44:57] If you are having access problems, please see: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [00:45:06] No Nova credentials found for your account. [00:45:10] I'm fairly sure I had some before [00:45:11] meh [00:45:11] #care [00:45:23] Reedy: Log in and out of the labs console [00:45:58] nope [00:46:14] Oh you're getting that on ssh? oof. [00:46:38] Also, lol http://www.spamex.com/ [00:53:55] PROBLEM Total Processes is now: CRITICAL on driver-dev3 driver-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:54:35] PROBLEM dpkg-check is now: CRITICAL on driver-dev3 driver-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [00:54:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [00:55:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [00:55:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [00:55:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [00:55:45] PROBLEM Current Load is now: CRITICAL on driver-dev3 driver-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:56:45] PROBLEM Current Users is now: CRITICAL on driver-dev3 driver-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:05] PROBLEM Disk Space is now: CRITICAL on driver-dev3 driver-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake. [00:57:15] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [00:58:05] PROBLEM Free ram is now: CRITICAL on driver-dev3 driver-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake. [01:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [01:24:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [01:25:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [01:25:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [01:25:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [01:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [01:32:05] RECOVERY Disk Space is now: OK on driver-dev3 driver-dev3 output: DISK OK [01:33:05] RECOVERY Free ram is now: OK on driver-dev3 driver-dev3 output: OK: 61% free memory [01:33:55] RECOVERY Total Processes is now: OK on driver-dev3 driver-dev3 output: PROCS OK: 130 processes [01:34:22] johnduhart: how is that different from spamgourmet? [01:35:12] jeremyb: It isn't, it's just funny that it hasn't been touched in so long. [01:35:45] RECOVERY Current Load is now: OK on driver-dev3 driver-dev3 output: OK - load average: 0.09, 0.26, 0.25 [01:35:56] And it's part of my new blacklist. [01:35:56] oh, wow [01:36:34] for mediawiki registrations? [01:36:45] RECOVERY Current Users is now: OK on driver-dev3 driver-dev3 output: USERS OK - 1 users currently logged in [01:37:01] jeremyb: Well the blacklist can be used on anything [01:37:24] why do you maintain one? [01:38:02] https://github.com/johnduhart/throwaway-email-blacklist/blob/master/domains.txt [01:38:44] I just started it. We'll use it for wikimedia but I want it to be usable for everyone [01:41:52] * jeremyb waves nimish_g. or no i don't you've /quit [01:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [01:54:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [01:55:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [01:55:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [01:55:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [01:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [02:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [02:24:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [02:25:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [02:25:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [02:25:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [02:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [02:40:35] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 20% free memory [02:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [02:54:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [02:55:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [02:55:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [02:55:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [02:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [02:58:55] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 18% free memory [03:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [03:24:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [03:25:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [03:25:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [03:25:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [03:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [03:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [03:54:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [03:55:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [03:55:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [03:55:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [03:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [04:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [04:24:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [04:25:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [04:25:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [04:25:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [04:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [04:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [04:54:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [04:55:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [04:55:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [04:55:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [04:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [05:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [05:24:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [05:25:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [05:25:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [05:25:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [05:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [05:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [05:54:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [05:55:25] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [05:55:25] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [05:55:35] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [05:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [06:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [06:24:55] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [06:25:31] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [06:25:38] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [06:25:38] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [06:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [06:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [06:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [06:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [06:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [06:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [06:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [07:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [07:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [07:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [07:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [07:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [07:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [07:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [07:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [07:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [07:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [07:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [07:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [08:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [08:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [08:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [08:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [08:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [08:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [08:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [08:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [08:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [08:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [08:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [08:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [09:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [09:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [09:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [09:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [09:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [09:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [09:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [09:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [09:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [09:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [09:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [09:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [10:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [10:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [10:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [10:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [10:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [10:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [10:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [10:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [10:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [10:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [10:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [10:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [11:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [11:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [11:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [11:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [11:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [11:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [11:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [11:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [11:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [11:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [11:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [11:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [12:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [12:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [12:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [12:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [12:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [12:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [12:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [12:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [12:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [12:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [12:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [12:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [13:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [13:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [13:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [13:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [13:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [13:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [13:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [13:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [13:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [13:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [13:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [13:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [14:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [14:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [14:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [14:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [14:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [14:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [14:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [14:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [14:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [14:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [14:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [14:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [15:08:55] RECOVERY Free ram is now: OK on bots-3 bots-3 output: OK: 39% free memory [15:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [15:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [15:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [15:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [15:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [15:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [15:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [15:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [15:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [15:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [15:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [15:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [16:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [16:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [16:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [16:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [16:26:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [16:27:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [16:54:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [16:55:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [16:55:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [16:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [16:56:05] PROBLEM host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [16:57:25] PROBLEM host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [17:19:07] johnduhart: hi [17:19:16] hi [17:19:20] want me to create an account on nagios for you? [17:19:29] you will be able to disable / enable alerts [17:19:36] etc [17:19:40] re your mail [17:20:14] That's okay, if you could just disable those alerts for now I'd appriciate it. [17:20:21] sure [17:20:30] I will create it anyway wait [17:23:55] it's in nagios.wmflabs.org/nlogin [17:24:36] ok [17:24:51] I think if you acknowledge it disable check too [17:24:53] but not sure about it [17:24:55] PROBLEM host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [17:25:05] PROBLEM host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [17:25:50] ACKNOWLEDGEMENT host: incubator-bots2 is DOWN address: incubator-bots2 CRITICAL - Host Unreachable (incubator-bots2) [17:25:55] PROBLEM host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [17:26:05] ACKNOWLEDGEMENT host: incubator-dep is DOWN address: incubator-dep CRITICAL - Host Unreachable (incubator-dep) [17:26:05] ACKNOWLEDGEMENT host: incubator-live is DOWN address: incubator-live CRITICAL - Host Unreachable (incubator-live) [17:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [17:26:20] ACKNOWLEDGEMENT host: incubator-testing is DOWN address: incubator-testing CRITICAL - Host Unreachable (incubator-testing) [17:26:20] ACKNOWLEDGEMENT host: incubator-nfs is DOWN address: incubator-nfs CRITICAL - Host Unreachable (incubator-nfs) [17:26:37] let's see [17:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [18:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [18:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [19:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [19:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [20:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [20:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [21:02:05] PROBLEM Free ram is now: WARNING on bots-3 bots-3 output: Warning: 19% free memory [21:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [21:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [22:25:25] PROBLEM Current Load is now: CRITICAL on bots-cb bots-cb output: CRITICAL - load average: 26.64, 38.04, 18.45 [22:26:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [22:30:25] PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 0.71, 14.21, 13.47 [22:50:25] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.30, 0.60, 3.97 [22:56:05] PROBLEM host: prefixexport is DOWN address: prefixexport CRITICAL - Host Unreachable (prefixexport) [23:00:26] Reedy: hi [23:00:39] you still need help on labs? [23:18:17] Omnomnomnom