[00:00:09] I'm probably mixing up accounts than [00:00:43] 2014/03/09 00:00 CRIT ha-dns-auth CRITICAL - Host Unreachable (ha-dns-auth) [00:00:43] 2014/03/09 00:00 CRIT ha-dns-recursor.esi CRITICAL - Host Unreachable (ha-dns-recursor.esi) [00:00:43] 2014/03/09 00:00 CRIT ha-sql.esi CRITICAL - Host Unreachable (ha-sql.esi) [00:01:43] 2014/03/08 23:54 CRIT adenia SMTP CRITICAL - Socket timeout after 10 seconds [00:01:43] 2014/03/08 23:54 CRIT hemlock /home CHECK_NRPE: Socket timeout after 30 seconds. [00:01:43] 2014/03/08 23:54 CRIT hemlock SMTP CRITICAL - Socket timeout after 10 seconds [00:01:43] 2014/03/08 23:54 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [00:01:43] 2014/03/08 23:55 CRIT nightshade SMTP CRITICAL - Socket timeout after 10 seconds [00:01:43] 2014/03/08 23:54 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [00:01:43] 2014/03/08 23:54 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [00:01:44] 2014/03/08 23:54 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [00:01:44] 2014/03/08 23:55 CRIT ptolemy SMTP CRITICAL - Socket timeout after 10 seconds [00:01:45] 2014/03/08 23:54 CRIT rosemary SMTP CRITICAL - Socket timeout after 10 seconds [00:01:45] 2014/03/08 23:55 CRIT sage SMTP CRITICAL - Socket timeout after 10 seconds [00:01:46] 2014/03/08 23:54 CRIT thyme SMTP CRITICAL - Socket timeout after 10 seconds [00:02:43] 2014/03/08 23:55 CRIT damiana DiskSuite CRITICAL - submirror d51 of mirror d50 is "Needs" and submirror d52 of mirror d50 is "Needs" and submirror d31 of mirror d30 is "Needs" and submirror d32 of mirror d30 is "Needs" and submirror d11 of mirror d10 is "Needs" and submirror d12 of mirror d10 is "Needs" and submirror d21 of mirror d20 is "Needs" and submirror d22 of mirror d20 is "Needs" [00:02:43] 2014/03/08 23:55 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [00:03:43] 2014/03/09 00:03 OK damiana DiskSuite OK - No disk failures detected [00:03:43] 2014/03/08 23:55 CRIT damiana NTP NTP CRITICAL: No response from NTP server [00:03:43] 2014/03/08 23:55 CRIT damiana SMTP CRITICAL - Socket timeout after 10 seconds [00:03:43] 2014/03/08 23:56 CRIT daphne SMTP CRITICAL - Socket timeout after 10 seconds [00:03:43] 2014/03/08 23:55 CRIT ha-nfs.esi NFS Connection refused [00:03:44] 2014/03/09 00:03 OK ortelius SMTP SMTP OK - 0.001 sec. response time [00:04:43] 2014/03/09 00:04 OK daphne SMTP SMTP OK - 0.012 sec. response time [00:04:43] 2014/03/09 00:03 OK ha-nfs.esi NFS TCP OK - 0.000 second response time on port 2049 [00:04:43] 2014/03/09 00:04 OK nightshade Sun Grid Engine execd Host and Queues Ok [00:04:43] 2014/03/09 00:03 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.006 second response time [00:04:43] 2014/03/09 00:04 OK willow SMTP SMTP OK - 5.008 sec. response time [00:04:44] 2014/03/09 00:03 WARN wolfsbane toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.009 second response time [00:05:43] 2014/03/09 00:04 OK ha-dns-auth PING OK - Packet loss = 0%, RTA = 0.17 ms [00:05:43] 2014/03/09 00:05 OK ha-dns-recursor.esi PING OK - Packet loss = 0%, RTA = 0.37 ms [00:05:43] 2014/03/09 00:04 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.36 ms [00:05:43] 2014/03/09 00:04 OK ha-proxy.esi PING OK - Packet loss = 0%, RTA = 0.21 ms [00:05:43] 2014/03/09 00:04 OK ha-sql.esi PING OK - Packet loss = 0%, RTA = 0.23 ms [00:06:54] 2014/03/09 00:06 OK nightshade Load avg. OK - load average: 8.66, 12.98, 8.54 [00:08:57] 2014/03/09 00:07 OK hemlock /home DISK OK - free space: /home 16266 MB (32% inode=91%): [00:08:57] 2014/03/09 00:07 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [00:08:57] 2014/03/09 00:07 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [00:15:54] 2014/03/09 00:15 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [00:17:54] 2014/03/09 00:17 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [00:21:54] 2014/03/09 00:15 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [00:58:55] 2014/03/09 00:52 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [00:59:56] 2014/03/09 00:59 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [01:14:57] 2014/03/09 01:08 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [01:16:58] 2014/03/09 01:10 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [01:18:58] 2014/03/09 01:18 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [01:18:58] 2014/03/09 01:18 OK nightshade Sun Grid Engine execd Host and Queues Ok [01:40:00] 2014/03/09 01:33 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [01:40:00] 2014/03/09 01:33 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [01:42:00] 2014/03/09 01:41 OK nightshade Sun Grid Engine execd Host and Queues Ok [01:46:00] 2014/03/09 01:45 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [01:53:02] 2014/03/09 01:46 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [01:53:02] 2014/03/09 01:46 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [01:57:02] 2014/03/09 01:56 OK nightshade Sun Grid Engine execd Host and Queues Ok [01:58:02] 2014/03/09 01:57 CRIT ha-dns-recursor.esi DNS recursor CRITICAL - Plugin timed out while executing system call [01:58:02] 2014/03/09 01:57 CRIT ha-dns-recursor.esi PING CRITICAL - Host Unreachable (ha-dns-recursor.esi) [01:59:02] 2014/03/09 01:58 CRIT damiana /tmp Timeout while attempting connection [01:59:02] 2014/03/09 01:58 CRIT damiana DiskSuite Connection refused or timed out [01:59:02] 2014/03/09 01:58 CRIT damiana Environment IPMI Connection refused or timed out [01:59:02] 2014/03/09 01:58 CRIT damiana Free Memory Connection refused or timed out [01:59:02] 2014/03/09 01:58 CRIT damiana Load avg. Connection refused or timed out [01:59:03] 2014/03/09 01:58 CRIT damiana ts-array5 Timeout while attempting connection [01:59:03] 2014/03/09 01:58 CRIT ha-dns-auth Authoritative DNS CRITICAL - Plugin timed out while executing system call [01:59:03] 2014/03/09 01:58 CRIT ha-dns-auth PING CRITICAL - Host Unreachable (ha-dns-auth) [01:59:04] 2014/03/09 01:58 CRIT ha-nfs.esi NFS No route to host [01:59:04] 2014/03/09 01:58 CRIT ha-nfs.esi PING CRITICAL - Host Unreachable (ha-nfs.esi) [01:59:05] 2014/03/09 01:57 CRIT ha-proxy.esi HTTP proxy CRITICAL - Socket timeout after 10 seconds [01:59:05] 2014/03/09 01:58 CRIT ha-proxy.esi PING CRITICAL - Host Unreachable (ha-proxy.esi) [02:00:03] 2014/03/09 01:59 CRIT damiana / Connection refused or timed out [02:00:03] 2014/03/09 01:58 CRIT damiana NTP CRITICAL - Socket timeout after 10 seconds [02:00:03] 2014/03/09 01:59 CRIT damiana PING CRITICAL - Host Unreachable (damiana) [02:00:03] 2014/03/09 01:59 CRIT damiana SMTP No route to host [02:00:03] 2014/03/09 01:59 CRIT damiana SSH No route to host [02:01:02] 2014/03/09 02:00 CRIT ha-dns-auth CRITICAL - Host Unreachable (ha-dns-auth) [02:01:02] 2014/03/09 02:00 CRIT ha-proxy.esi CRITICAL - Host Unreachable (ha-proxy.esi) [02:01:02] 2014/03/09 02:00 OK damiana Load avg. OK - load average: 1.84, 0.64, 0.23 [02:01:02] 2014/03/09 02:00 OK ha-nfs.esi PING PING OK - Packet loss = 0%, RTA = 0.21 ms [02:02:02] 2014/03/09 02:01 CRIT ha-sql.esi CRITICAL - Host Unreachable (ha-sql.esi) [02:02:02] 2014/03/09 02:01 OK damiana / DISK OK - free space: / 27872 MB (38% inode=95%): [02:02:02] 2014/03/09 02:01 OK damiana /tmp DISK OK - free space: /tmp 6300 MB (99% inode=99%): [02:02:02] 2014/03/09 02:01 OK damiana Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [02:02:02] 2014/03/09 02:01 OK damiana PING PING OK - Packet loss = 0%, RTA = 0.15 ms [02:02:03] 2014/03/09 02:01 OK damiana SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:02:03] 2014/03/09 02:01 OK damiana ts-array5 2/2 paths are active [02:03:02] 2014/03/09 02:02 CRIT ha-dns-recursor.esi CRITICAL - Host Unreachable (ha-dns-recursor.esi) [02:04:02] 2014/03/09 01:57 CRIT adenia SMTP CRITICAL - Socket timeout after 10 seconds [02:04:03] 2014/03/09 01:56 CRIT clematis SMTP CRITICAL - Socket timeout after 10 seconds [02:04:03] 2014/03/09 01:57 CRIT daphne SMTP CRITICAL - Socket timeout after 10 seconds [02:04:03] 2014/03/09 01:57 CRIT hemlock SMTP CRITICAL - Socket timeout after 10 seconds [02:04:03] 2014/03/09 01:56 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [02:04:03] 2014/03/09 01:57 CRIT nightshade SMTP CRITICAL - Socket timeout after 10 seconds [02:04:03] 2014/03/09 01:57 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [02:04:03] 2014/03/09 01:57 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [02:04:04] 2014/03/09 01:57 CRIT ptolemy SMTP CRITICAL - Socket timeout after 10 seconds [02:04:05] 2014/03/09 01:56 CRIT rosemary SMTP CRITICAL - Socket timeout after 10 seconds [02:04:05] 2014/03/09 01:57 CRIT sage SMTP CRITICAL - Socket timeout after 10 seconds [02:04:05] 2014/03/09 01:57 CRIT thyme SMTP CRITICAL - Socket timeout after 10 seconds [02:05:03] 2014/03/09 01:57 CRIT hemlock /home CHECK_NRPE: Socket timeout after 30 seconds. [02:05:03] 2014/03/09 01:57 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [02:05:03] 2014/03/09 01:57 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [02:06:03] 2014/03/09 02:05 ?? adenia SMTP check_smtp: Invalid hostname/address - adenia [02:06:03] 2014/03/09 01:58 CRIT damiana NTP NTP CRITICAL: No response from NTP server [02:06:03] 2014/03/09 01:59 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [02:07:03] 2014/03/09 01:58 CRIT damiana DiskSuite CRITICAL - submirror d51 of mirror d50 is "Needs" and submirror d52 of mirror d50 is "Needs" and submirror d31 of mirror d30 is "Needs" and submirror d32 of mirror d30 is "Needs" and submirror d11 of mirror d10 is "Needs" and submirror d12 of mirror d10 is "Needs" and submirror d21 of mirror d20 is "Needs" and submirror d22 of mirror d20 is "Needs" [02:07:03] 2014/03/09 01:58 CRIT ha-nfs.esi NFS Connection refused [02:07:03] 2014/03/09 02:06 ?? hemlock SMTP check_smtp: Invalid hostname/address - hemlock [02:07:03] 2014/03/09 02:06 ?? thyme s1 replag check_mysql_query: Invalid hostname/address - thyme [02:07:03] 2014/03/09 02:06 WARN wolfsbane toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.011 second response time [02:08:03] 2014/03/09 02:07 OK damiana DiskSuite OK - No disk failures detected [02:08:03] 2014/03/09 02:07 OK ha-nfs.esi NFS TCP OK - 0.000 second response time on port 2049 [02:08:03] 2014/03/09 02:07 ?? hyacinth SMTP check_smtp: Invalid hostname/address - hyacinth [02:08:03] 2014/03/09 02:07 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [02:08:03] 2014/03/09 02:06 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.013 second response time [02:08:04] 2014/03/09 02:07 ?? thyme SMTP check_smtp: Invalid hostname/address - thyme [02:09:03] 2014/03/09 02:08 OK ha-dns-recursor.esi PING OK - Packet loss = 0%, RTA = 0.32 ms [02:09:03] 2014/03/09 02:08 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.20 ms [02:09:03] 2014/03/09 02:08 OK ha-proxy.esi PING OK - Packet loss = 0%, RTA = 0.25 ms [02:09:03] 2014/03/09 02:08 OK clematis SMTP SMTP OK - 0.003 sec. response time [02:09:03] 2014/03/09 02:08 OK damiana NTP NTP OK: Offset -0.000952 secs [02:09:04] 2014/03/09 02:08 ?? daphne SMTP check_smtp: Invalid hostname/address - daphne [02:09:04] 2014/03/09 02:08 ?? daphne wikidata replag check_mysql_query: Invalid hostname/address - daphne [02:09:05] 2014/03/09 02:08 OK ha-dns-recursor.esi PING PING OK - Packet loss = 0%, RTA = 0.39 ms [02:09:05] 2014/03/09 02:08 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.15 ms [02:09:06] 2014/03/09 02:08 OK ha-proxy.esi HTTP proxy HTTP OK: HTTP/1.0 302 Moved Temporarily - 649 bytes in 0.237 second response time [02:09:06] 2014/03/09 02:08 ?? ha-sql.esi MySQL check_mysql: Invalid hostname/address - ha-sql.esi [02:09:07] 2014/03/09 02:08 OK nightshade Sun Grid Engine execd Host and Queues Ok [02:09:07] 2014/03/09 02:08 OK ortelius SMTP SMTP OK - 0.002 sec. response time [02:09:07] 2014/03/09 02:08 OK rosemary SMTP SMTP OK - 0.003 sec. response time [02:10:03] 2014/03/09 02:09 OK ha-dns-auth PING OK - Packet loss = 0%, RTA = 0.44 ms [02:10:03] 2014/03/09 02:09 OK ha-sql.esi PING OK - Packet loss = 0%, RTA = 0.32 ms [02:10:03] 2014/03/09 02:09 OK adenia SMTP SMTP OK - 0.020 sec. response time [02:10:03] 2014/03/09 02:09 OK daphne SMTP SMTP OK - 0.021 sec. response time [02:10:03] 2014/03/09 02:09 CRIT daphne wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 18191111.000000 [02:10:04] 2014/03/09 02:09 OK ha-dns-auth Authoritative DNS DNS OK: 0.062 seconds response time. 1.www.toolserver.org returns 91.198.174.203 [02:10:04] 2014/03/09 02:09 OK ha-dns-auth PING PING OK - Packet loss = 0%, RTA = 0.48 ms [02:10:05] 2014/03/09 02:09 OK ha-dns-recursor.esi DNS recursor DNS OK: 0.186 seconds response time. www.google.com returns 173.194.46.112,173.194.46.113,173.194.46.114,173.194.46.115,173.194.46.116 [02:10:05] 2014/03/09 02:09 OK ha-proxy.esi PING PING OK - Packet loss = 0%, RTA = 0.17 ms [02:10:06] 2014/03/09 02:09 CRIT ha-sql.esi MySQL Access denied for user 'tsnagios7643'@'turnera' (using password: YES) [02:10:06] 2014/03/09 02:09 OK ha-sql.esi PING PING OK - Packet loss = 0%, RTA = 0.23 ms [02:10:07] 2014/03/09 02:09 OK hemlock SMTP SMTP OK - 0.022 sec. response time [02:10:07] 2014/03/09 02:09 OK hyacinth SMTP SMTP OK - 0.183 sec. response time [02:10:08] 2014/03/09 02:09 OK nightshade SMTP SMTP OK - 0.002 sec. response time [02:20:04] 2014/03/09 02:19 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [02:22:03] 2014/03/09 02:21 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [02:26:03] 2014/03/09 02:19 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [03:12:07] 2014/03/09 03:11 OK ortelius / DISK OK - free space: / 7425 MB (24% inode=91%): [03:18:07] 2014/03/09 03:11 CRIT ptolemy SMF ERROR - maintenance: svc:/network/ts/apache22:default [03:26:08] 2014/03/09 03:25 CRIT ha-proxy.esi HTTP proxy CRITICAL - Socket timeout after 10 seconds [03:26:08] 2014/03/09 03:25 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [03:27:08] 2014/03/09 03:26 CRIT damiana Free Memory Timeout while attempting connection [03:27:08] 2014/03/09 03:26 CRIT damiana NTP CRITICAL - Socket timeout after 10 seconds [03:27:08] 2014/03/09 03:26 CRIT ha-dns-auth Authoritative DNS CRITICAL - Plugin timed out while executing system call [03:27:08] 2014/03/09 03:26 CRIT ha-dns-auth PING CRITICAL - Host Unreachable (ha-dns-auth) [03:27:08] 2014/03/09 03:26 CRIT ha-dns-recursor.esi DNS recursor CRITICAL - Plugin timed out while executing system call [03:27:09] 2014/03/09 03:26 CRIT ha-dns-recursor.esi PING CRITICAL - Host Unreachable (ha-dns-recursor.esi) [03:27:09] 2014/03/09 03:26 CRIT ha-proxy.esi PING CRITICAL - Host Unreachable (ha-proxy.esi) [03:27:10] 2014/03/09 03:26 CRIT ha-sql.esi PING CRITICAL - Host Unreachable (ha-sql.esi) [03:27:10] 2014/03/09 03:25 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [03:28:08] 2014/03/09 03:27 CRIT damiana / Connection refused or timed out [03:28:08] 2014/03/09 03:27 CRIT damiana /tmp Connection refused or timed out [03:28:08] 2014/03/09 03:27 CRIT damiana DiskSuite Connection refused or timed out [03:28:08] 2014/03/09 03:27 CRIT damiana Environment IPMI Connection refused or timed out [03:28:08] 2014/03/09 03:26 CRIT damiana Load avg. Connection refused or timed out [03:28:09] 2014/03/09 03:27 CRIT damiana PING CRITICAL - Host Unreachable (damiana) [03:28:09] 2014/03/09 03:27 CRIT damiana SMTP CRITICAL - Socket timeout after 10 seconds [03:28:10] 2014/03/09 03:27 CRIT damiana SSH CRITICAL - Socket timeout after 10 seconds [03:28:10] 2014/03/09 03:27 CRIT damiana ts-array5 Connection refused or timed out [03:28:11] 2014/03/09 03:27 CRIT ha-nfs.esi NFS No route to host [03:28:11] 2014/03/09 03:27 CRIT ha-nfs.esi PING CRITICAL - Host Unreachable (ha-nfs.esi) [03:29:08] 2014/03/09 03:28 CRIT ha-dns-recursor.esi CRITICAL - Host Unreachable (ha-dns-recursor.esi) [03:29:08] 2014/03/09 03:28 CRIT ha-proxy.esi CRITICAL - Host Unreachable (ha-proxy.esi) [03:29:08] 2014/03/09 03:29 CRIT ha-sql.esi CRITICAL - Host Unreachable (ha-sql.esi) [03:29:08] 2014/03/09 03:29 OK damiana PING PING OK - Packet loss = 0%, RTA = 0.20 ms [03:29:08] 2014/03/09 03:28 OK ha-nfs.esi PING PING OK - Packet loss = 0%, RTA = 0.18 ms [03:30:08] 2014/03/09 03:29 CRIT ha-dns-auth CRITICAL - Host Unreachable (ha-dns-auth) [03:30:08] 2014/03/09 03:29 OK damiana / DISK OK - free space: / 27631 MB (38% inode=95%): [03:30:08] 2014/03/09 03:29 OK damiana /tmp DISK OK - free space: /tmp 6309 MB (99% inode=99%): [03:30:08] 2014/03/09 03:29 OK damiana Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [03:30:08] 2014/03/09 03:29 OK damiana Free Memory OK - 88.2% (7390340 kB) free. [03:30:09] 2014/03/09 03:29 OK damiana Load avg. OK - load average: 1.46, 0.73, 0.29 [03:30:09] 2014/03/09 03:29 OK damiana SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:30:10] 2014/03/09 03:29 OK damiana ts-array5 2/2 paths are active [03:32:09] 2014/03/09 03:25 CRIT adenia SMTP CRITICAL - Socket timeout after 10 seconds [03:32:09] 2014/03/09 03:25 CRIT clematis SMTP CRITICAL - Socket timeout after 10 seconds [03:32:09] 2014/03/09 03:25 CRIT daphne SMTP CRITICAL - Socket timeout after 10 seconds [03:32:09] 2014/03/09 03:25 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [03:32:09] 2014/03/09 03:25 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [03:32:09] 2014/03/09 03:25 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [03:32:10] 2014/03/09 03:25 CRIT ptolemy SMTP CRITICAL - Socket timeout after 10 seconds [03:32:10] 2014/03/09 03:25 CRIT rosemary SMTP CRITICAL - Socket timeout after 10 seconds [03:32:11] 2014/03/09 03:25 CRIT sage SMTP CRITICAL - Socket timeout after 10 seconds [03:32:11] 2014/03/09 03:25 CRIT thyme SMTP CRITICAL - Socket timeout after 10 seconds [03:32:12] 2014/03/09 03:25 CRIT wolfsbane SMTP CRITICAL - Socket timeout after 10 seconds [03:32:12] 2014/03/09 03:25 CRIT z-dat-s1-b SMTP CRITICAL - Socket timeout after 10 seconds [03:32:13] 2014/03/09 03:25 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [03:32:13] 2014/03/09 03:25 CRIT z-dat-s5-b SMTP CRITICAL - Socket timeout after 10 seconds [03:33:09] 2014/03/09 03:26 CRIT nightshade SMTP CRITICAL - Socket timeout after 10 seconds [03:33:09] 2014/03/09 03:25 CRIT yarrow SMTP CRITICAL - Socket timeout after 10 seconds [03:34:09] 2014/03/09 03:27 CRIT damiana DiskSuite CRITICAL - submirror d51 of mirror d50 is "Needs" and submirror d52 of mirror d50 is "Needs" and submirror d31 of mirror d30 is "Needs" and submirror d32 of mirror d30 is "Needs" and submirror d11 of mirror d10 is "Needs" and submirror d12 of mirror d10 is "Needs" and submirror d21 of mirror d20 is "Needs" and submirror d22 of mirror d20 is "Needs" [03:34:09] 2014/03/09 03:27 CRIT ha-nfs.esi NFS Connection refused [03:34:09] 2014/03/09 03:27 CRIT ptolemy PostgreSQL CRITICAL - database osm_mapnik (10 sec.) [03:34:09] 2014/03/09 03:27 CRIT willow SMTP CRITICAL - Socket timeout after 10 seconds [03:35:09] 2014/03/09 03:26 CRIT damiana NTP NTP CRITICAL: No response from NTP server [03:35:09] 2014/03/09 03:28 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [03:36:09] 2014/03/09 03:35 OK damiana DiskSuite OK - No disk failures detected [03:36:09] 2014/03/09 03:35 OK daphne SMTP SMTP OK - 5.017 sec. response time [03:36:09] 2014/03/09 03:35 OK ha-nfs.esi NFS TCP OK - 0.001 second response time on port 2049 [03:36:09] 2014/03/09 03:35 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.004 second response time [03:36:09] 2014/03/09 03:35 WARN wolfsbane toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.003 second response time [03:37:09] 2014/03/09 03:36 OK ha-dns-auth PING OK - Packet loss = 0%, RTA = 0.17 ms [03:37:09] 2014/03/09 03:36 OK ha-dns-recursor.esi PING OK - Packet loss = 0%, RTA = 0.32 ms [03:37:09] 2014/03/09 03:36 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.28 ms [03:37:09] 2014/03/09 03:36 OK ha-proxy.esi PING OK - Packet loss = 0%, RTA = 0.14 ms [03:37:09] 2014/03/09 03:36 OK ha-sql.esi PING OK - Packet loss = 0%, RTA = 0.35 ms [03:37:10] 2014/03/09 03:36 OK adenia SMTP SMTP OK - 0.003 sec. response time [03:37:10] 2014/03/09 03:36 OK clematis SMTP SMTP OK - 0.003 sec. response time [03:37:11] 2014/03/09 03:36 OK damiana NTP NTP OK: Offset -0.000823 secs [03:37:11] 2014/03/09 03:37 OK ha-dns-auth Authoritative DNS DNS OK: 0.214 seconds response time. 1.www.toolserver.org returns 91.198.174.203 [03:37:12] 2014/03/09 03:36 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.19 ms [03:37:12] 2014/03/09 03:36 OK ha-proxy.esi HTTP proxy HTTP OK: HTTP/1.0 302 Moved Temporarily - 1247 bytes in 0.250 second response time [03:37:13] 2014/03/09 03:37 OK ha-proxy.esi PING PING OK - Packet loss = 0%, RTA = 0.21 ms [03:37:13] 2014/03/09 03:36 OK hyacinth SMTP SMTP OK - 0.003 sec. response time [03:37:14] 2014/03/09 03:37 OK nightshade SMTP SMTP OK - 0.002 sec. response time [03:38:09] 2014/03/09 03:37 OK ha-dns-auth PING PING OK - Packet loss = 0%, RTA = 0.21 ms [03:38:09] 2014/03/09 03:37 OK ha-dns-recursor.esi DNS recursor DNS OK: 0.026 seconds response time. www.google.com returns 173.194.46.112,173.194.46.113,173.194.46.114,173.194.46.115,173.194.46.116 [03:38:09] 2014/03/09 03:37 OK ha-dns-recursor.esi PING PING OK - Packet loss = 0%, RTA = 0.19 ms [03:38:09] 2014/03/09 03:37 OK ha-sql.esi PING PING OK - Packet loss = 0%, RTA = 0.33 ms [03:38:09] 2014/03/09 03:37 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [03:38:10] 2014/03/09 03:37 OK nightshade aliasd TCP OK - 2.520 second response time on port 984 [500 Not found.] [03:38:10] 2014/03/09 03:37 OK ptolemy SMTP SMTP OK - 0.004 sec. response time [03:38:11] 2014/03/09 03:37 OK sage SMTP SMTP OK - 0.002 sec. response time [03:38:11] 2014/03/09 03:37 OK willow SMTP SMTP OK - 0.003 sec. response time [03:38:12] 2014/03/09 03:37 OK z-dat-s5-b SMTP SMTP OK - 0.003 sec. response time [03:48:10] 2014/03/09 03:47 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [03:49:10] 2014/03/09 03:49 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [03:54:09] 2014/03/09 03:47 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [04:47:12] 2014/03/09 04:40 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [04:49:12] 2014/03/09 04:42 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [04:52:12] 2014/03/09 04:51 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [04:52:13] 2014/03/09 04:51 OK nightshade Sun Grid Engine execd Host and Queues Ok [05:04:12] 2014/03/09 05:03 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.40 ms [05:04:12] 2014/03/09 05:03 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.18 ms [05:04:13] 2014/03/09 04:57 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [05:04:13] 2014/03/09 05:03 OK nightshade aliasd TCP OK - 5.014 second response time on port 984 [500 Not found.] [05:05:13] 2014/03/09 05:04 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [05:15:12] 2014/03/09 05:14 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [05:16:12] 2014/03/09 05:15 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [05:20:12] 2014/03/09 05:13 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [05:25:12] 2014/03/09 05:25 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.30 ms [05:26:12] 2014/03/09 05:25 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.16 ms [05:26:13] 2014/03/09 05:25 OK nightshade aliasd TCP OK - 1.008 second response time on port 984 [500 Not found.] [05:31:12] 2014/03/09 05:30 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [05:33:12] 2014/03/09 05:32 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [05:37:13] 2014/03/09 05:30 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [06:27:14] 2014/03/09 06:20 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [06:28:14] 2014/03/09 06:27 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [06:35:14] 2014/03/09 06:28 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [06:35:14] 2014/03/09 06:28 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [06:39:15] 2014/03/09 06:38 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [06:39:16] 2014/03/09 06:38 OK nightshade Sun Grid Engine execd Host and Queues Ok [06:50:15] 2014/03/09 06:43 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [06:50:15] 2014/03/09 06:43 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [06:53:17] 2014/03/09 06:52 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [06:53:17] 2014/03/09 06:52 OK nightshade Sun Grid Engine execd Host and Queues Ok [07:03:16] 2014/03/09 07:02 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.54 ms [07:03:16] 2014/03/09 07:02 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.21 ms [07:03:16] 2014/03/09 07:02 OK nightshade aliasd TCP OK - 1.008 second response time on port 984 [500 Not found.] [07:14:17] 2014/03/09 07:13 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [07:15:18] 2014/03/09 07:14 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [07:19:18] 2014/03/09 07:12 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [07:40:04] Marlen Caemmerer * [Toolserver-l] Toolserver still down [07:43:22] 2014/03/09 07:43 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.42 ms [07:44:22] 2014/03/09 07:43 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.17 ms [07:44:22] 2014/03/09 07:43 OK nightshade aliasd TCP OK - 0.003 second response time on port 984 [500 Not found.] [07:44:22] 2014/03/09 07:43 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [07:44:22] 2014/03/09 07:43 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [07:45:22] 2014/03/09 07:44 CRIT damiana / Timeout while attempting connection [07:45:22] 2014/03/09 07:44 CRIT damiana /tmp Timeout while attempting connection [07:45:22] 2014/03/09 07:44 CRIT damiana DiskSuite Timeout while attempting connection [07:45:22] 2014/03/09 07:44 CRIT damiana Environment IPMI Connection refused or timed out [07:45:22] 2014/03/09 07:44 CRIT damiana Free Memory Connection refused or timed out [07:45:22] 2014/03/09 07:44 CRIT damiana Load avg. Connection refused or timed out [07:45:22] 2014/03/09 07:44 CRIT damiana NTP CRITICAL - Socket timeout after 10 seconds [07:45:23] 2014/03/09 07:45 CRIT damiana PING CRITICAL - Host Unreachable (damiana) [07:45:23] 2014/03/09 07:44 CRIT damiana ts-array5 Timeout while attempting connection [07:45:24] 2014/03/09 07:45 CRIT ha-dns-auth Authoritative DNS CRITICAL - Plugin timed out while executing system call [07:45:24] 2014/03/09 07:45 CRIT ha-dns-auth PING CRITICAL - Host Unreachable (ha-dns-auth) [07:45:25] 2014/03/09 07:44 CRIT ha-dns-recursor.esi DNS recursor CRITICAL - Plugin timed out while executing system call [07:45:25] 2014/03/09 07:44 CRIT ha-dns-recursor.esi PING CRITICAL - Host Unreachable (ha-dns-recursor.esi) [07:45:26] 2014/03/09 07:44 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [07:46:21] 2014/03/09 07:46 CRIT damiana CRITICAL - Host Unreachable (damiana) [07:46:21] 2014/03/09 07:45 CRIT damiana SMTP Connection refused [07:46:21] 2014/03/09 07:45 CRIT damiana SSH Connection refused [07:47:21] 2014/03/09 07:46 OK damiana PING OK - Packet loss = 0%, RTA = 0.37 ms [07:47:21] 2014/03/09 07:47 CRIT ha-dns-recursor.esi CRITICAL - Host Unreachable (ha-dns-recursor.esi) [07:47:22] 2014/03/09 07:46 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [07:47:22] 2014/03/09 07:47 CRIT ha-proxy.esi CRITICAL - Host Unreachable (ha-proxy.esi) [07:47:22] 2014/03/09 07:47 OK damiana / DISK OK - free space: / 27695 MB (38% inode=95%): [07:47:22] 2014/03/09 07:46 OK damiana Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [07:47:22] 2014/03/09 07:46 OK damiana Free Memory OK - 89.6% (7509404 kB) free. [07:47:23] 2014/03/09 07:46 OK damiana Load avg. OK - load average: 1.71, 0.70, 0.26 [07:47:23] 2014/03/09 07:47 OK damiana PING PING OK - Packet loss = 0%, RTA = 0.19 ms [07:47:24] 2014/03/09 07:47 OK damiana SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:47:24] 2014/03/09 07:46 OK ha-nfs.esi PING PING OK - Packet loss = 0%, RTA = 0.36 ms [07:48:21] 2014/03/09 07:48 CRIT ha-sql.esi CRITICAL - Host Unreachable (ha-sql.esi) [07:49:22] 2014/03/09 07:48 CRIT ha-dns-auth CRITICAL - Host Unreachable (ha-dns-auth) [07:49:22] 2014/03/09 07:48 ?? z-dat-s5-b s4 replag check_mysql_query: Invalid hostname/address - z-dat-s5-b [07:50:22] 2014/03/09 07:44 CRIT nightshade SMTP CRITICAL - Socket timeout after 10 seconds [07:50:22] 2014/03/09 07:43 CRIT rosemary SMTP CRITICAL - Socket timeout after 10 seconds [07:50:22] 2014/03/09 07:43 CRIT yarrow SMTP CRITICAL - Socket timeout after 10 seconds [07:50:22] 2014/03/09 07:47 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [07:50:22] 2014/03/09 07:49 CRIT z-dat-s5-b s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 821133.000000 [07:51:22] 2014/03/09 07:44 CRIT adenia SMTP CRITICAL - Socket timeout after 10 seconds [07:51:22] 2014/03/09 07:44 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [07:51:22] 2014/03/09 07:44 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [07:51:22] 2014/03/09 07:44 CRIT nightshade aliasd Connection refused [07:51:22] 2014/03/09 07:44 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [07:51:22] 2014/03/09 07:44 CRIT ptolemy SMTP CRITICAL - Socket timeout after 10 seconds [07:51:22] 2014/03/09 07:44 CRIT sage SMTP CRITICAL - Socket timeout after 10 seconds [07:51:23] 2014/03/09 07:44 CRIT thyme SMTP CRITICAL - Socket timeout after 10 seconds [07:51:23] 2014/03/09 07:44 CRIT willow SMTP CRITICAL - Socket timeout after 10 seconds [07:51:24] 2014/03/09 07:44 CRIT wolfsbane SMTP CRITICAL - Socket timeout after 10 seconds [07:51:24] 2014/03/09 07:44 CRIT z-dat-s1-b SMTP CRITICAL - Socket timeout after 10 seconds [07:51:25] 2014/03/09 07:50 CRIT z-dat-s5-b SMTP CRITICAL - Socket timeout after 10 seconds [07:52:22] 2014/03/09 07:44 CRIT damiana NTP NTP CRITICAL: No response from NTP server [07:52:22] 2014/03/09 07:51 OK willow SMTP SMTP OK - 3.188 sec. response time [07:53:22] 2014/03/09 07:45 CRIT damiana SMTP CRITICAL - Socket timeout after 10 seconds [07:53:22] 2014/03/09 07:52 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.003 second response time [07:53:22] 2014/03/09 07:52 WARN wolfsbane toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.006 second response time [07:54:22] 2014/03/09 07:54 OK ha-dns-auth PING OK - Packet loss = 0%, RTA = 0.21 ms [07:54:22] 2014/03/09 07:54 OK ha-dns-recursor.esi PING OK - Packet loss = 0%, RTA = 0.17 ms [07:54:22] 2014/03/09 07:54 OK ha-proxy.esi PING OK - Packet loss = 0%, RTA = 0.34 ms [07:54:22] 2014/03/09 07:54 OK damiana SMTP SMTP OK - 0.003 sec. response time [07:54:22] 2014/03/09 07:54 OK ha-dns-auth Authoritative DNS DNS OK: 4.527 seconds response time. 1.www.toolserver.org returns 91.198.174.203 [07:54:22] 2014/03/09 07:54 OK ha-dns-auth PING PING OK - Packet loss = 0%, RTA = 0.17 ms [07:54:22] 2014/03/09 07:54 WARN ha-proxy.esi PING PING WARNING - Packet loss = 0%, RTA = 304.80 ms [07:54:23] 2014/03/09 07:54 OK nightshade SMTP SMTP OK - 0.010 sec. response time [07:54:23] 2014/03/09 07:54 OK ptolemy SMTP SMTP OK - 0.004 sec. response time [07:54:24] 2014/03/09 07:53 OK yarrow SMTP SMTP OK - 5.012 sec. response time [07:54:24] 2014/03/09 07:54 OK z-dat-s5-b SMTP SMTP OK - 0.012 sec. response time [07:55:22] 2014/03/09 07:54 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.31 ms [07:55:22] 2014/03/09 07:54 OK ha-sql.esi PING OK - Packet loss = 0%, RTA = 0.17 ms [07:55:22] 2014/03/09 07:54 OK adenia SMTP SMTP OK - 0.004 sec. response time [07:55:22] 2014/03/09 07:54 OK damiana NTP NTP OK: Offset 0.061333 secs [07:55:22] 2014/03/09 07:54 OK ha-dns-recursor.esi DNS recursor DNS OK: 0.318 seconds response time. www.google.com returns 173.194.46.112,173.194.46.113,173.194.46.114,173.194.46.115,173.194.46.116 [07:55:22] 2014/03/09 07:54 OK ha-dns-recursor.esi PING PING OK - Packet loss = 0%, RTA = 0.28 ms [07:55:22] 2014/03/09 07:54 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.19 ms [07:55:23] 2014/03/09 07:54 OK ha-proxy.esi HTTP proxy HTTP OK: HTTP/1.0 302 Moved Temporarily - 1247 bytes in 0.314 second response time [07:55:23] 2014/03/09 07:55 OK ha-proxy.esi PING PING OK - Packet loss = 0%, RTA = 0.17 ms [07:55:24] 2014/03/09 07:54 OK ha-sql.esi PING PING OK - Packet loss = 0%, RTA = 0.18 ms [07:55:24] 2014/03/09 07:54 OK hyacinth SMTP SMTP OK - 0.003 sec. response time [07:55:25] 2014/03/09 07:54 OK nightshade Sun Grid Engine execd Host and Queues Ok [07:57:22] 2014/03/09 07:56 OK nightshade aliasd TCP OK - 1.009 second response time on port 984 [500 Not found.] [08:05:23] 2014/03/09 08:04 CRIT ha-ldap.esi PING PING CRITICAL - Packet loss = 100% [08:07:23] 2014/03/09 08:06 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [08:11:25] 2014/03/09 08:04 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [08:55:26] 2014/03/09 08:48 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [08:57:26] 2014/03/09 08:57 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [09:37:27] 2014/03/09 09:30 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [09:37:27] 2014/03/09 09:30 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [09:38:26] 2014/03/09 09:37 OK nightshade Sun Grid Engine execd Host and Queues Ok [09:46:28] 2014/03/09 09:39 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [09:47:30] 2014/03/09 09:47 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [09:47:30] 2014/03/09 09:47 OK nightshade Sun Grid Engine execd Host and Queues Ok [09:51:29] 2014/03/09 09:50 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.51 ms [09:51:29] 2014/03/09 09:50 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.16 ms [09:51:29] 2014/03/09 09:50 OK nightshade aliasd TCP OK - 4.005 second response time on port 984 [500 Not found.] [10:05:48] amette nosy1 good morning, is it still broken? [10:05:54] yes [10:07:07] any eta? [10:07:30] 2014/03/09 10:06 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [10:07:30] 2014/03/09 10:01 WARN hawthorn NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 0.162865 secs [10:08:31] 2014/03/09 10:08 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [10:13:31] 2014/03/09 10:06 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [10:13:49] Danny_B|webgate: after-fix? ;) [10:13:58] sorry, dont know [10:19:59] no prob, just curious [10:36:33] 2014/03/09 10:36 OK hawthorn NTP NTP OK: Offset 0.1095 secs [10:43:32] 2014/03/09 10:42 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.49 ms [10:43:32] 2014/03/09 10:42 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.19 ms [10:44:32] 2014/03/09 10:43 OK nightshade aliasd TCP OK - 0.003 second response time on port 984 [500 Not found.] [10:49:32] 2014/03/09 10:48 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [10:50:32] 2014/03/09 10:50 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [10:55:33] 2014/03/09 10:48 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [11:28:34] 2014/03/09 11:28 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.28 ms [11:29:34] 2014/03/09 11:28 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.14 ms [11:29:34] 2014/03/09 11:28 OK nightshade aliasd TCP OK - 1.009 second response time on port 984 [500 Not found.] [11:30:34] 2014/03/09 11:30 CRIT ha-sql.esi PING CRITICAL - Host Unreachable (ha-sql.esi) [11:30:34] 2014/03/09 11:29 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [11:31:34] 2014/03/09 11:30 CRIT damiana / Connection refused or timed out [11:31:34] 2014/03/09 11:30 CRIT damiana /tmp Timeout while attempting connection [11:31:34] 2014/03/09 11:30 CRIT damiana DiskSuite Timeout while attempting connection [11:31:34] 2014/03/09 11:30 CRIT damiana Environment IPMI Timeout while attempting connection [11:31:34] 2014/03/09 11:30 CRIT damiana Free Memory Timeout while attempting connection [11:31:34] 2014/03/09 11:30 CRIT damiana Load avg. Connection refused or timed out [11:31:35] 2014/03/09 11:30 CRIT damiana NTP CRITICAL - Socket timeout after 10 seconds [11:31:36] 2014/03/09 11:31 CRIT damiana PING CRITICAL - Host Unreachable (damiana) [11:31:36] 2014/03/09 11:31 CRIT damiana SMTP No route to host [11:31:37] 2014/03/09 11:31 CRIT damiana SSH No route to host [11:31:37] 2014/03/09 11:30 CRIT damiana ts-array5 Timeout while attempting connection [11:31:38] 2014/03/09 11:31 CRIT ha-dns-auth Authoritative DNS CRITICAL - Plugin timed out while executing system call [11:31:38] 2014/03/09 11:31 CRIT ha-dns-auth PING CRITICAL - Host Unreachable (ha-dns-auth) [11:31:38] 2014/03/09 11:30 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [11:32:35] 2014/03/09 11:32 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [11:32:35] 2014/03/09 11:31 CRIT ha-dns-recursor.esi DNS recursor CRITICAL - Plugin timed out while executing system call [11:32:35] 2014/03/09 11:31 CRIT ha-dns-recursor.esi PING CRITICAL - Host Unreachable (ha-dns-recursor.esi) [11:33:35] 2014/03/09 11:32 CRIT ha-dns-recursor.esi CRITICAL - Host Unreachable (ha-dns-recursor.esi) [11:33:35] 2014/03/09 11:33 CRIT ha-proxy.esi CRITICAL - Host Unreachable (ha-proxy.esi) [11:33:35] 2014/03/09 11:33 OK damiana / DISK OK - free space: / 27154 MB (37% inode=95%): [11:33:35] 2014/03/09 11:32 OK damiana Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [11:33:35] 2014/03/09 11:32 OK damiana Free Memory OK - 89.2% (7473664 kB) free. [11:33:35] 2014/03/09 11:32 OK damiana Load avg. OK - load average: 1.75, 0.70, 0.26 [11:33:36] 2014/03/09 11:33 OK damiana PING PING OK - Packet loss = 0%, RTA = 0.17 ms [11:33:36] 2014/03/09 11:33 OK damiana SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:33:37] 2014/03/09 11:32 OK ha-nfs.esi PING PING OK - Packet loss = 0%, RTA = 0.39 ms [11:34:35] 2014/03/09 11:34 CRIT ha-dns-auth CRITICAL - Host Unreachable (ha-dns-auth) [11:34:35] 2014/03/09 11:33 CRIT ha-sql.esi CRITICAL - Host Unreachable (ha-sql.esi) [11:36:35] 2014/03/09 11:30 CRIT nightshade SMTP CRITICAL - Socket timeout after 10 seconds [11:36:35] 2014/03/09 11:30 CRIT ptolemy SMTP CRITICAL - Socket timeout after 10 seconds [11:36:35] 2014/03/09 11:29 CRIT yarrow SMTP CRITICAL - Socket timeout after 10 seconds [11:36:35] 2014/03/09 11:30 CRIT z-dat-s5-b SMTP CRITICAL - Socket timeout after 10 seconds [11:37:35] 2014/03/09 11:30 CRIT adenia SMTP CRITICAL - Socket timeout after 10 seconds [11:37:35] 2014/03/09 11:30 CRIT hemlock SMTP CRITICAL - Socket timeout after 10 seconds [11:37:35] 2014/03/09 11:30 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [11:37:35] 2014/03/09 11:30 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [11:37:35] 2014/03/09 11:30 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [11:37:35] 2014/03/09 11:30 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [11:37:36] 2014/03/09 11:30 CRIT rosemary SMTP CRITICAL - Socket timeout after 10 seconds [11:37:36] 2014/03/09 11:30 CRIT sage SMTP CRITICAL - Socket timeout after 10 seconds [11:37:37] 2014/03/09 11:30 CRIT thyme SMTP CRITICAL - Socket timeout after 10 seconds [11:37:37] 2014/03/09 11:30 CRIT willow SMTP CRITICAL - Socket timeout after 10 seconds [11:37:38] 2014/03/09 11:30 CRIT z-dat-s1-b SMTP CRITICAL - Socket timeout after 10 seconds [11:37:38] 2014/03/09 11:30 CRIT z-dat-s2-b SMTP CRITICAL - Socket timeout after 10 seconds [11:38:35] 2014/03/09 11:30 CRIT damiana NTP NTP CRITICAL: No response from NTP server [11:38:35] 2014/03/09 11:31 CRIT damiana SMTP CRITICAL - Socket timeout after 10 seconds [11:38:35] 2014/03/09 11:37 OK hemlock SMTP SMTP OK - 9.018 sec. response time [11:39:35] 2014/03/09 11:38 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.005 second response time [11:39:35] 2014/03/09 11:38 WARN wolfsbane toolserver.org HTTP HTTP WARNING: HTTP/1.1 404 Not found - 161 bytes in 0.011 second response time [11:39:35] 2014/03/09 11:39 ?? z-dat-s5-b SMTP check_smtp: Invalid hostname/address - z-dat-s5-b [11:39:35] 2014/03/09 11:38 ?? z-dat-s5-b s4 replag check_mysql_query: Invalid hostname/address - z-dat-s5-b [11:40:35] 2014/03/09 11:40 OK ha-dns-auth PING OK - Packet loss = 0%, RTA = 0.17 ms [11:40:35] 2014/03/09 11:40 OK ha-dns-recursor.esi PING OK - Packet loss = 0%, RTA = 0.33 ms [11:40:35] 2014/03/09 11:40 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.35 ms [11:40:35] 2014/03/09 11:40 OK ha-proxy.esi PING OK - Packet loss = 0%, RTA = 0.41 ms [11:40:36] 2014/03/09 11:40 OK ha-sql.esi PING OK - Packet loss = 0%, RTA = 0.24 ms [11:40:36] 2014/03/09 11:40 OK damiana SMTP SMTP OK - 0.003 sec. response time [11:40:36] 2014/03/09 11:40 OK ha-dns-auth Authoritative DNS DNS OK: 5.465 seconds response time. 1.www.toolserver.org returns 91.198.174.203 [11:40:36] 2014/03/09 11:40 OK ha-dns-auth PING PING OK - Packet loss = 0%, RTA = 0.26 ms [11:40:37] 2014/03/09 11:40 OK ha-dns-recursor.esi DNS recursor DNS OK: 0.134 seconds response time. www.google.com returns 173.194.46.112,173.194.46.113,173.194.46.114,173.194.46.115,173.194.46.116 [11:40:38] 2014/03/09 11:40 WARN ha-proxy.esi PING PING WARNING - Packet loss = 0%, RTA = 259.98 ms [11:40:38] 2014/03/09 11:40 OK ha-sql.esi PING PING OK - Packet loss = 0%, RTA = 0.29 ms [11:40:38] 2014/03/09 11:40 OK nightshade SMTP SMTP OK - 5.016 sec. response time [11:41:36] 2014/03/09 11:40 OK adenia SMTP SMTP OK - 0.008 sec. response time [11:41:36] 2014/03/09 11:40 OK damiana NTP NTP OK: Offset 0.053879 secs [11:41:36] 2014/03/09 11:40 OK ha-dns-recursor.esi PING PING OK - Packet loss = 0%, RTA = 0.24 ms [11:41:36] 2014/03/09 11:40 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.31 ms [11:41:36] 2014/03/09 11:40 OK ha-proxy.esi HTTP proxy HTTP OK: HTTP/1.0 302 Moved Temporarily - 1247 bytes in 0.304 second response time [11:41:36] 2014/03/09 11:41 OK ha-proxy.esi PING PING OK - Packet loss = 0%, RTA = 0.21 ms [11:41:37] 2014/03/09 11:40 OK hyacinth SMTP SMTP OK - 0.002 sec. response time [11:41:38] 2014/03/09 11:40 OK nightshade aliasd TCP OK - 1.136 second response time on port 984 [500 Not found.] [11:41:38] 2014/03/09 11:40 OK ortelius SMTP SMTP OK - 0.002 sec. response time [11:41:39] 2014/03/09 11:40 OK rosemary SMTP SMTP OK - 0.002 sec. response time [11:41:39] 2014/03/09 11:40 OK thyme SMTP SMTP OK - 0.014 sec. response time [11:41:40] 2014/03/09 11:40 OK wolfsbane SMTP SMTP OK - 0.002 sec. response time [11:41:40] 2014/03/09 11:40 OK z-dat-s1-b SMTP SMTP OK - 0.003 sec. response time [11:42:36] 2014/03/09 11:42 OK z-dat-s5-b SMTP SMTP OK - 0.014 sec. response time [11:42:36] 2014/03/09 11:41 CRIT z-dat-s5-b s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 835027.000000 [11:48:14] amette / nosy1 : Still working? [11:48:42] The maps on the English and French Wikipedia stopped working [11:52:36] 2014/03/09 11:51 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [11:53:36] 2014/03/09 11:52 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [11:57:37] 2014/03/09 11:50 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [11:58:12] multichill: yes still on it [12:04:38] 2014/03/09 12:03 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.30 ms [12:04:39] 2014/03/09 12:04 OK nightshade aliasd TCP OK - 1.009 second response time on port 984 [500 Not found.] [12:05:38] 2014/03/09 12:04 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.19 ms [12:10:38] 2014/03/09 12:09 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [12:11:38] 2014/03/09 12:11 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [12:16:40] 2014/03/09 12:16 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.29 ms [12:17:40] 2014/03/09 12:16 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.14 ms [12:21:42] 2014/03/09 12:21 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [12:23:41] 2014/03/09 12:23 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [12:28:42] 2014/03/09 12:21 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [12:33:42] 2014/03/09 12:33 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.30 ms [12:33:42] 2014/03/09 12:33 OK nightshade aliasd TCP OK - 1.007 second response time on port 984 [500 Not found.] [12:34:42] 2014/03/09 12:33 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.17 ms [12:40:43] 2014/03/09 12:39 CRIT ha-ldap.esi PING CRITICAL - Host Unreachable (ha-ldap.esi) [12:41:43] 2014/03/09 12:40 CRIT ha-ldap.esi CRITICAL - Host Unreachable (ha-ldap.esi) [12:45:42] 2014/03/09 12:38 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [13:23:45] 2014/03/09 13:17 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [13:25:45] 2014/03/09 13:25 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [13:35:47] 2014/03/09 13:29 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [13:36:47] 2014/03/09 13:29 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [13:39:47] 2014/03/09 13:39 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [13:39:47] 2014/03/09 13:39 OK nightshade Sun Grid Engine execd Host and Queues Ok [14:09:51] 2014/03/09 14:03 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [14:13:52] 2014/03/09 14:07 ?? nightshade Sun Grid Engine execd Execution timeout exceeded [14:17:52] 2014/03/09 14:17 OK ha-ldap.esi PING OK - Packet loss = 0%, RTA = 0.92 ms [14:17:52] 2014/03/09 14:17 OK ha-ldap.esi PING PING OK - Packet loss = 0%, RTA = 0.16 ms [14:17:52] 2014/03/09 14:17 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [14:17:52] 2014/03/09 14:17 OK nightshade Sun Grid Engine execd Host and Queues Ok [14:17:52] 2014/03/09 14:17 OK nightshade aliasd TCP OK - 1.008 second response time on port 984 [500 Not found.] [14:47:53] 2014/03/09 14:41 CRIT z-dat-s2-b SSH CRITICAL - Socket timeout after 10 seconds [16:36:04] 2014/03/09 16:29 WARN hemlock / DISK WARNING - free space: / 4190 MB (20% inode=88%): [16:36:04] 2014/03/09 16:29 WARN hemlock /tmp DISK WARNING - free space: / 4190 MB (20% inode=88%): [20:27:58] amette : Any news? [20:33:09] 2014/03/09 20:33 OK z-dat-s2-b SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [20:59:30] amette: Any status updates? [21:16:04] Maarten Dammers * Re: [Toolserver-l] Toolserver still down [21:17:10] hoo: Doesn't seem to respond. I send an email to the list ^^ [21:17:30] Just saw that [21:22:32] Feels like a deja-vu anyhow. I'm quite sure last year we were promised that Toolserver had to be down for a week or so so that the Solaris LDAP server could be replaced with a Linux one that would be easiert to maintain. And now the same situation again? [21:43:16] 2014/03/09 21:42 OK hemlock /home DISK OK - free space: /home 16254 MB (32% inode=91%): [21:43:16] 2014/03/09 21:42 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.010 second response time [21:43:16] 2014/03/09 21:42 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [21:57:16] 2014/03/09 21:56 OK ha-ldap.esi LDAP LDAP OK - 0.004 seconds response time [21:57:17] 2014/03/09 21:56 CRIT hemlock.mgmt PING CRITICAL - Host Unreachable (hemlock.mgmt) [21:57:17] 2014/03/09 21:56 CRIT hemlock.mgmt SSH No route to host [21:57:17] 2014/03/09 21:56 CRIT rosemary.mgmt PING CRITICAL - Host Unreachable (rosemary.mgmt) [21:57:17] 2014/03/09 21:56 CRIT rosemary.mgmt SSH No route to host [21:57:17] 2014/03/09 21:56 CRIT willow.mgmt PING CRITICAL - Host Unreachable (willow.mgmt) [21:57:17] 2014/03/09 21:56 CRIT willow.mgmt SSH No route to host [21:57:18] 2014/03/09 21:56 CRIT yarrow.mgmt PING CRITICAL - Host Unreachable (yarrow.mgmt) [21:57:19] 2014/03/09 21:56 CRIT yarrow.mgmt SSH No route to host [21:58:16] 2014/03/09 21:57 CRIT nightshade.mgmt PING CRITICAL - Host Unreachable (nightshade.mgmt) [21:58:17] 2014/03/09 21:57 CRIT nightshade.mgmt SSH No route to host [22:02:17] 2014/03/09 22:01 CRIT hemlock.mgmt CRITICAL - Host Unreachable (hemlock.mgmt) [22:02:17] 2014/03/09 22:01 CRIT nightshade.mgmt CRITICAL - Host Unreachable (nightshade.mgmt) [22:02:17] 2014/03/09 22:01 CRIT rosemary.mgmt CRITICAL - Host Unreachable (rosemary.mgmt) [22:02:17] 2014/03/09 22:01 CRIT willow.mgmt CRITICAL - Host Unreachable (willow.mgmt) [22:02:17] 2014/03/09 22:01 CRIT yarrow.mgmt CRITICAL - Host Unreachable (yarrow.mgmt) [22:13:04] Marlen Caemmerer * Re: [Toolserver-l] Toolserver still down [22:27:17] 2014/03/09 22:26 OK ptolemy SMF OK - all services online [22:30:18] 2014/03/09 22:29 OK yarrow aliasd TCP OK - 0.001 second response time on port 984 [500 Not found.]