[01:38:26] 2014/03/10 01:30 CRIT cassia s4 replag (Service Check Timed Out) [01:47:26] 2014/03/10 01:45 OK cassia s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 0.000000 [01:50:27] 2014/03/10 01:44 ?? cassia Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:51:27] 2014/03/10 01:44 ?? cassia / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:51:27] 2014/03/10 01:44 ?? cassia /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:51:27] 2014/03/10 01:44 ?? cassia /sql/data/dewiki CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:51:27] 2014/03/10 01:44 ?? cassia /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:51:27] 2014/03/10 01:44 ?? cassia Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:53:27] 2014/03/10 01:52 OK cassia / DISK OK - free space: / 11874 MB (59% inode=91%): [01:53:27] 2014/03/10 01:52 OK cassia /sql DISK OK - free space: /sql 654199 MB (55% inode=99%): [01:53:27] 2014/03/10 01:52 OK cassia /sql/data/dewiki DISK OK - free space: /sql 654199 MB (55% inode=99%): [01:53:27] 2014/03/10 01:52 OK cassia /tmp DISK OK - free space: /tmp 126 MB (99% inode=99%): [01:53:27] 2014/03/10 01:52 WARN cassia Environment IPMI NRPE: Unable to read output [01:54:27] 2014/03/10 01:54 OK cassia Load avg. OK - load average: 0.14, 0.18, 0.25 [01:55:27] 2014/03/10 01:54 ?? cassia Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:56:28] 2014/03/10 01:55 CRIT cassia Environment IPMI Connection refused by host [01:59:29] 2014/03/10 01:58 OK cassia Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [02:06:29] 2014/03/10 02:03 ?? cassia Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:07:29] 2014/03/10 02:03 ?? cassia / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:07:29] 2014/03/10 02:03 ?? cassia /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:07:29] 2014/03/10 02:03 ?? cassia /sql/data/dewiki CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:07:29] 2014/03/10 02:03 ?? cassia /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:07:29] 2014/03/10 02:03 ?? cassia Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:10:31] 2014/03/10 02:09 OK cassia /tmp DISK OK - free space: /tmp 131 MB (99% inode=99%): [02:10:31] 2014/03/10 02:09 OK cassia Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [02:10:31] 2014/03/10 02:10 OK cassia Load avg. OK - load average: 0.20, 0.16, 0.19 [02:12:31] 2014/03/10 02:11 OK cassia / DISK OK - free space: / 11874 MB (59% inode=91%): [02:12:31] 2014/03/10 02:11 OK cassia /sql DISK OK - free space: /sql 654171 MB (55% inode=99%): [02:12:31] 2014/03/10 02:11 OK cassia /sql/data/dewiki DISK OK - free space: /sql 654171 MB (55% inode=99%): [02:21:32] 2014/03/10 02:14 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [02:41:33] 2014/03/10 02:35 ?? cassia Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:42:33] 2014/03/10 02:35 ?? cassia / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:42:33] 2014/03/10 02:35 ?? cassia /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:42:33] 2014/03/10 02:35 ?? cassia /sql/data/dewiki CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:42:33] 2014/03/10 02:35 ?? cassia /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:42:33] 2014/03/10 02:35 ?? cassia Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:08:32] 2014/03/10 02:37 ?? cassia RAID CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:46:33] 2014/03/10 03:45 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3405.000000 [03:57:32] 2014/03/10 03:56 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1796.000000 [05:04:36] 2014/03/10 04:57 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [06:00:36] 2014/03/10 06:00 CRIT cassia Load avg. Connection refused by host [06:01:36] 2014/03/10 06:00 CRIT cassia / Connection refused by host [06:01:36] 2014/03/10 06:00 CRIT cassia /sql Connection refused by host [06:01:36] 2014/03/10 06:00 CRIT cassia /sql/data/dewiki Connection refused by host [06:01:36] 2014/03/10 06:00 CRIT cassia /tmp Connection refused by host [06:01:36] 2014/03/10 06:00 CRIT cassia Environment IPMI Connection refused by host [06:03:36] 2014/03/10 06:02 CRIT cassia RAID Connection refused by host [06:05:36] 2014/03/10 06:04 OK cassia /sql/data/dewiki DISK OK - free space: /sql 653787 MB (55% inode=99%): [06:05:36] 2014/03/10 06:04 OK cassia /tmp DISK OK - free space: /tmp 140 MB (99% inode=99%): [06:05:37] 2014/03/10 06:04 OK cassia Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [06:05:37] 2014/03/10 06:05 OK cassia Load avg. OK - load average: 0.20, 0.15, 0.15 [06:06:36] 2014/03/10 06:05 OK cassia / DISK OK - free space: / 11874 MB (59% inode=91%): [06:06:37] 2014/03/10 06:05 OK cassia /sql DISK OK - free space: /sql 653787 MB (55% inode=99%): [06:08:37] 2014/03/10 06:07 OK cassia RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [06:41:38] 2014/03/10 06:35 ?? cassia / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:41:39] 2014/03/10 06:35 ?? cassia /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:42:38] 2014/03/10 06:35 ?? cassia /sql/data/dewiki CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:42:39] 2014/03/10 06:35 ?? cassia /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:42:39] 2014/03/10 06:35 ?? cassia Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:42:39] 2014/03/10 06:36 ?? cassia Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:08:39] 2014/03/10 06:37 ?? cassia RAID CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:21:47] 2014/03/10 08:15 WARN ptolemy /sql DISK WARNING - free space: /sql 169558 MB (20% inode=99%): [09:22:48] 2014/03/10 09:22 OK cassia / DISK OK - free space: / 11873 MB (59% inode=91%): [09:22:49] 2014/03/10 09:22 OK cassia /sql DISK OK - free space: /sql 654461 MB (55% inode=99%): [09:22:49] 2014/03/10 09:22 OK cassia /sql/data/dewiki DISK OK - free space: /sql 654461 MB (55% inode=99%): [09:23:49] 2014/03/10 09:22 OK cassia /tmp DISK OK - free space: /tmp 141 MB (99% inode=99%): [09:23:49] 2014/03/10 09:22 OK cassia Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [09:23:49] 2014/03/10 09:23 OK cassia Load avg. OK - load average: 0.21, 0.31, 0.41 [09:23:49] 2014/03/10 09:22 OK cassia RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:08:51] 2014/03/10 09:37 ?? cassia RAID CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:37:54] 2014/03/10 10:36 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [10:40:55] 2014/03/10 10:39 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [11:23:56] 2014/03/10 11:16 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [11:24:56] 2014/03/10 11:17 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [11:39:56] 2014/03/10 11:38 CRIT ortelius /tmp Timeout while attempting connection [11:39:56] 2014/03/10 11:39 CRIT ortelius DiskSuite Timeout while attempting connection [11:39:56] 2014/03/10 11:39 CRIT ortelius Environment IPMI Timeout while attempting connection [11:39:56] 2014/03/10 11:39 CRIT ortelius Load avg. Timeout while attempting connection [11:39:56] 2014/03/10 11:39 CRIT ortelius NTP CRITICAL - Socket timeout after 10 seconds [11:39:56] 2014/03/10 11:39 CRIT ortelius PING PING CRITICAL - Packet loss = 100% [11:39:57] 2014/03/10 11:39 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [11:39:57] 2014/03/10 11:38 CRIT ortelius Sun Grid Engine execd Timeout while attempting connection [11:39:58] 2014/03/10 11:38 CRIT wolfsbane Sun Grid Engine execd Connection refused by host [11:40:56] 2014/03/10 11:39 CRIT ortelius / Connection refused or timed out [11:40:56] 2014/03/10 11:39 CRIT ortelius SSH CRITICAL - Socket timeout after 10 seconds [11:40:56] 2014/03/10 11:39 CRIT wolfsbane /tmp Timeout while attempting connection [11:40:56] 2014/03/10 11:40 CRIT wolfsbane Environment IPMI Timeout while attempting connection [11:40:56] 2014/03/10 11:40 CRIT wolfsbane Load avg. Connection refused or timed out [11:40:56] 2014/03/10 11:40 CRIT wolfsbane NTP CRITICAL - Socket timeout after 10 seconds [11:40:57] 2014/03/10 11:40 CRIT wolfsbane PING CRITICAL - Host Unreachable (wolfsbane) [11:40:57] 2014/03/10 11:40 CRIT wolfsbane SMTP No route to host [11:41:56] 2014/03/10 11:41 OK ortelius PING PING OK - Packet loss = 0%, RTA = 0.11 ms [11:41:56] 2014/03/10 11:41 OK ortelius SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:41:56] 2014/03/10 11:40 CRIT wolfsbane / Connection refused or timed out [11:41:56] 2014/03/10 11:40 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [11:42:56] 2014/03/10 11:41 OK ortelius / DISK OK - free space: / 11366 MB (37% inode=91%): [11:42:56] 2014/03/10 11:41 OK ortelius /tmp DISK OK - free space: /tmp 25538 MB (99% inode=99%): [11:42:56] 2014/03/10 11:42 OK ortelius DiskSuite OK - No disk failures detected [11:42:56] 2014/03/10 11:42 OK ortelius Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [11:42:56] 2014/03/10 11:42 OK ortelius Load avg. OK - load average: 3.79, 1.38, 0.52 [11:42:57] 2014/03/10 11:42 OK ortelius NTP NTP OK: Offset 0.156556 secs [11:42:57] 2014/03/10 11:42 OK ortelius SMTP SMTP OK - 2.165 sec. response time [11:42:58] 2014/03/10 11:41 WARN ortelius Sun Grid Engine execd NRPE: Unable to read output [11:42:58] 2014/03/10 11:41 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [11:42:59] 2014/03/10 11:42 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [11:42:59] 2014/03/10 11:42 OK wolfsbane Load avg. OK - load average: 2.19, 0.71, 0.25 [11:43:00] 2014/03/10 11:42 OK wolfsbane PING PING OK - Packet loss = 0%, RTA = 0.21 ms [11:43:00] 2014/03/10 11:42 OK wolfsbane SMTP SMTP OK - 0.140 sec. response time [11:43:01] 2014/03/10 11:42 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:43:56] 2014/03/10 11:42 OK wolfsbane / DISK OK - free space: / 11628 MB (38% inode=93%): [11:43:56] 2014/03/10 11:42 OK wolfsbane /tmp DISK OK - free space: / 11627 MB (38% inode=93%): [11:43:56] 2014/03/10 11:43 OK wolfsbane NTP NTP OK: Offset 0.000747 secs [11:43:56] 2014/03/10 11:42 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.003 second response time [12:22:59] 2014/03/10 12:22 OK cassia / DISK OK - free space: / 11873 MB (59% inode=91%): [12:22:59] 2014/03/10 12:22 OK cassia /sql DISK OK - free space: /sql 654167 MB (55% inode=99%): [12:22:59] 2014/03/10 12:22 OK cassia /sql/data/dewiki DISK OK - free space: /sql 654167 MB (55% inode=99%): [12:22:59] 2014/03/10 12:22 OK cassia /tmp DISK OK - free space: /tmp 141 MB (99% inode=99%): [12:23:59] 2014/03/10 12:22 OK cassia Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [12:23:59] 2014/03/10 12:23 OK cassia Load avg. OK - load average: 0.18, 0.21, 0.21 [12:23:59] 2014/03/10 12:22 OK cassia RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:42:01] 2014/03/10 12:35 CRIT cassia / Connection refused by host [12:42:01] 2014/03/10 12:35 CRIT cassia /sql Connection refused by host [12:42:01] 2014/03/10 12:35 CRIT cassia /sql/data/dewiki Connection refused by host [12:42:01] 2014/03/10 12:35 CRIT cassia /tmp Connection refused by host [12:42:01] 2014/03/10 12:35 CRIT cassia Environment IPMI Connection refused by host [12:43:01] 2014/03/10 12:36 CRIT cassia Load avg. Connection refused by host [12:50:02] 2014/03/10 12:42 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [13:01:03] 2014/03/10 13:00 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.006 second response time [13:09:02] 2014/03/10 12:37 CRIT cassia RAID Connection refused by host [14:03:07] 2014/03/10 14:02 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:07:07] 2014/03/10 14:06 CRIT wolfsbane Sun Grid Engine execd Connection refused by host [14:07:07] 2014/03/10 13:59 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [14:08:07] 2014/03/10 14:07 CRIT wolfsbane Environment IPMI Connection refused by host [14:08:07] 2014/03/10 14:07 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:09:07] 2014/03/10 14:08 ?? wolfsbane Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:10:08] 2014/03/10 14:02 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [14:10:08] 2014/03/10 14:09 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [14:10:08] 2014/03/10 14:09 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:12:08] 2014/03/10 14:11 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.014 second response time [14:16:08] 2014/03/10 14:15 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:18:08] 2014/03/10 14:16 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:20:08] 2014/03/10 14:19 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:27:12] 2014/03/10 14:26 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:28:09] 2014/03/10 14:21 ?? wolfsbane /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:28:09] 2014/03/10 14:27 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:29:10] 2014/03/10 14:22 ?? wolfsbane Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:31:10] 2014/03/10 14:24 ?? wolfsbane Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:32:10] 2014/03/10 14:25 ?? wolfsbane / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:33:10] 2014/03/10 14:32 OK wolfsbane / DISK OK - free space: / 10395 MB (34% inode=93%): [14:33:10] 2014/03/10 14:32 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:33:10] 2014/03/10 14:32 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.009 second response time [14:34:10] 2014/03/10 14:33 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [14:34:10] 2014/03/10 14:33 OK wolfsbane Load avg. OK - load average: 0.20, 0.12, 0.28 [14:34:11] 2014/03/10 14:33 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:36:11] 2014/03/10 14:35 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:37:11] 2014/03/10 14:36 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:39:11] 2014/03/10 14:38 OK wolfsbane /tmp DISK OK - free space: / 10394 MB (34% inode=93%): [14:40:11] 2014/03/10 14:33 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [14:49:31] 2014/03/10 14:41 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:49:32] 2014/03/10 14:43 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:49:37] 2014/03/10 14:42 ?? wolfsbane Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:50:12] 2014/03/10 14:43 ?? wolfsbane Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:50:12] 2014/03/10 14:49 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:51:12] 2014/03/10 14:50 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [14:51:12] 2014/03/10 14:50 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:52:12] 2014/03/10 14:51 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:53:13] 2014/03/10 14:52 OK wolfsbane Load avg. OK - load average: 0.15, 0.12, 0.16 [14:53:13] 2014/03/10 14:52 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:55:13] 2014/03/10 14:54 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [14:58:14] 2014/03/10 14:57 ?? wolfsbane Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:01:14] 2014/03/10 15:00 ?? wolfsbane Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:04:14] 2014/03/10 14:57 ?? wolfsbane Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:05:14] 2014/03/10 15:04 CRIT wolfsbane Environment IPMI Connection refused by host [15:05:15] 2014/03/10 15:04 CRIT wolfsbane Load avg. Connection refused by host [15:05:15] 2014/03/10 14:58 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [15:05:15] 2014/03/10 15:04 CRIT wolfsbane Sun Grid Engine execd Connection refused by host [15:06:14] 2014/03/10 15:04 CRIT wolfsbane /tmp Connection refused by host [15:09:15] 2014/03/10 15:04 CRIT wolfsbane / Connection refused by host [15:10:15] 2014/03/10 15:09 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:17:16] 2014/03/10 15:10 CRIT wolfsbane SMTP Connection refused [15:17:16] 2014/03/10 15:10 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [15:27:18] 2014/03/10 15:26 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:39:19] 2014/03/10 15:32 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [16:03:21] 2014/03/10 16:02 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:10:21] 2014/03/10 16:03 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [16:22:23] 2014/03/10 16:15 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.595 second response time [16:23:23] 2014/03/10 16:22 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.006 second response time [16:50:25] 2014/03/10 16:43 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [16:56:25] 2014/03/10 16:55 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [17:22:26] 2014/03/10 17:15 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [17:23:25] 2014/03/10 17:22 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.046 second response time [18:25:28] 2014/03/10 18:24 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:41:29] 2014/03/10 18:34 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [18:45:29] 2014/03/10 18:44 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time [18:53:29] 2014/03/10 18:46 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [18:54:29] 2014/03/10 18:47 CRIT hyacinth RAID Timeout while attempting connection [18:55:29] 2014/03/10 18:49 CRIT hyacinth / Timeout while attempting connection [18:55:30] 2014/03/10 18:48 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [18:55:30] 2014/03/10 18:48 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [18:56:30] 2014/03/10 18:56 OK hyacinth / DISK OK - free space: / 8480 MB (28% inode=86%): [18:56:31] 2014/03/10 18:49 CRIT hyacinth Environment IPMI Timeout while attempting connection [18:56:31] 2014/03/10 18:55 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [18:56:31] 2014/03/10 18:48 CRIT z-dat-s4-a MySQL Can't connect to MySQL server on 'z-dat-s4-a' (110) [18:56:31] 2014/03/10 18:48 CRIT z-dat-s4-a MySQL slave Can't connect to MySQL server on 'z-dat-s4-a' (110) [18:56:31] 2014/03/10 18:48 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [18:56:31] 2014/03/10 18:48 CRIT z-dat-s6-a MySQL slave Can't connect to MySQL server on 'z-dat-s6-a' (110) [18:56:32] 2014/03/10 18:48 CRIT z-dat-s7-a wikidata replag (Service Check Timed Out) [18:57:30] 2014/03/10 18:56 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [18:57:31] 2014/03/10 18:56 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:57:31] 2014/03/10 18:56 OK hyacinth SMTP SMTP OK - 0.002 sec. response time [18:57:31] 2014/03/10 18:56 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:57:31] 2014/03/10 18:49 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [18:57:31] 2014/03/10 18:56 OK z-dat-s4-a MySQL Uptime: 2437906 Threads: 1 Questions: 309674445 Slow queries: 116 Opens: 295 Flush tables: 1 Open tables: 263 Queries per second avg: 127.24 [18:57:31] 2014/03/10 18:56 OK z-dat-s4-a MySQL slave Uptime: 2437917 Threads: 1 Questions: 309678373 Slow queries: 116 Opens: 295 Flush tables: 1 Open tables: 263 Queries per second avg: 127.25 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 419 [18:57:32] 2014/03/10 18:56 OK z-dat-s6-a MySQL Uptime: 211577 Threads: 8 Questions: 39075864 Slow queries: 17728 Opens: 1263 Flush tables: 1 Open tables: 1116 Queries per second avg: 184.688 [18:57:32] 2014/03/10 18:56 OK z-dat-s6-a MySQL slave Uptime: 211577 Threads: 8 Questions: 39075864 Slow queries: 17728 Opens: 1263 Flush tables: 1 Open tables: 1116 Queries per second avg: 184.688 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 471 [18:57:33] 2014/03/10 18:56 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 419.000000 [18:58:31] 2014/03/10 18:57 OK z-dat-s3-a MySQL Uptime: 530031 Threads: 13 Questions: 198763465 Slow queries: 7094 Opens: 258510 Flush tables: 1 Open tables: 16384 Queries per second avg: 375.3 [19:12:32] 2014/03/10 19:05 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [20:11:39] 2014/03/10 20:11 OK hawthorn SMTP SMTP OK - 0.002 sec. response time [20:31:41] 2014/03/10 20:24 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [20:31:46] Toolserver is 502'ing for me :-/ [20:34:41] 2014/03/10 20:33 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time [20:37:17] twkozlowski: Which URL? [20:38:28] scfc_de: GeoHack tool, http://toolserver.org/~geohack/geohack.php?language=pl&pagename=Wroc%C5%82aw_Miko%C5%82aj%C3%B3w¶ms=51_06_55_N_16_59_53_E_type:city [20:38:31] but it's fine now [20:57:41] 2014/03/10 20:56 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:07:43] 2014/03/10 21:00 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [21:11:44] 2014/03/10 21:11 CRIT hawthorn SMTP CRITICAL - Socket timeout after 10 seconds [21:11:44] 2014/03/10 21:11 CRIT hawthorn SSH No route to host [21:11:44] 2014/03/10 21:11 CRIT hawthorn Sun Grid Engine execd Connection refused or timed out [21:12:44] 2014/03/10 21:11 CRIT hawthorn / Connection refused or timed out [21:12:44] 2014/03/10 21:11 CRIT hawthorn /tmp Connection refused or timed out [21:12:44] 2014/03/10 21:11 CRIT hawthorn Environment IPMI Connection refused or timed out [21:12:44] 2014/03/10 21:11 CRIT hawthorn Load avg. Connection refused or timed out [21:12:44] 2014/03/10 21:12 CRIT hawthorn NTP CRITICAL - Socket timeout after 10 seconds [21:12:44] 2014/03/10 21:12 CRIT hawthorn PING CRITICAL - Host Unreachable (hawthorn) [21:13:44] 2014/03/10 21:13 OK hawthorn / DISK OK - free space: / 18183 MB (60% inode=93%): [21:13:44] 2014/03/10 21:13 OK hawthorn /tmp DISK OK - free space: /tmp 6444 MB (100% inode=99%): [21:13:44] 2014/03/10 21:13 OK hawthorn PING PING OK - Packet loss = 0%, RTA = 0.25 ms [21:13:44] 2014/03/10 21:13 OK hawthorn SMTP SMTP OK - 0.687 sec. response time [21:13:44] 2014/03/10 21:13 OK hawthorn SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:14:44] 2014/03/10 21:13 OK hawthorn Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [21:14:44] 2014/03/10 21:13 OK hawthorn Load avg. OK - load average: 1.06, 0.61, 0.24 [21:14:44] 2014/03/10 21:14 OK hawthorn NTP NTP OK: Offset -0.039392 secs [21:14:44] 2014/03/10 21:13 ?? hawthorn Sun Grid Engine execd option -w not recognized [21:22:44] 2014/03/10 21:15 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.653 second response time [21:23:44] 2014/03/10 21:22 CRIT ortelius Sun Grid Engine execd Connection refused by host [21:24:44] 2014/03/10 21:24 CRIT clematis / Timeout while attempting connection [21:24:44] 2014/03/10 21:24 CRIT clematis /tmp Timeout while attempting connection [21:24:44] 2014/03/10 21:24 CRIT clematis Environment IPMI Connection refused or timed out [21:24:44] 2014/03/10 21:24 CRIT clematis Load avg. Connection refused or timed out [21:24:44] 2014/03/10 21:24 CRIT clematis Sun Grid Engine execd Timeout while attempting connection [21:25:44] 2014/03/10 21:24 CRIT clematis NTP CRITICAL - Socket timeout after 10 seconds [21:25:44] 2014/03/10 21:24 CRIT clematis PING CRITICAL - Host Unreachable (clematis) [21:25:44] 2014/03/10 21:24 CRIT clematis SMTP No route to host [21:25:44] 2014/03/10 21:25 CRIT clematis SSH No route to host [21:25:44] 2014/03/10 21:24 CRIT ortelius / Timeout while attempting connection [21:25:45] 2014/03/10 21:24 CRIT ortelius /tmp Timeout while attempting connection [21:25:45] 2014/03/10 21:25 CRIT ortelius DiskSuite Timeout while attempting connection [21:25:46] 2014/03/10 21:25 CRIT ortelius Environment IPMI Timeout while attempting connection [21:25:46] 2014/03/10 21:25 CRIT ortelius NTP CRITICAL - Socket timeout after 10 seconds [21:26:44] 2014/03/10 21:26 OK clematis / DISK OK - free space: / 18746 MB (62% inode=93%): [21:26:44] 2014/03/10 21:26 OK clematis /tmp DISK OK - free space: /tmp 6526 MB (100% inode=99%): [21:26:44] 2014/03/10 21:26 OK clematis Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [21:26:44] 2014/03/10 21:26 OK clematis Load avg. OK - load average: 1.53, 0.62, 0.23 [21:26:44] 2014/03/10 21:26 WARN clematis Sun Grid Engine execd option -w not recognized [21:26:45] 2014/03/10 21:25 CRIT ortelius Load avg. Timeout while attempting connection [21:26:45] 2014/03/10 21:25 CRIT ortelius PING PING CRITICAL - Packet loss = 100% [21:26:46] 2014/03/10 21:25 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [21:26:46] 2014/03/10 21:25 CRIT ortelius SSH CRITICAL - Socket timeout after 10 seconds [21:27:44] 2014/03/10 21:26 OK clematis NTP NTP OK: Offset -0.000413 secs [21:27:44] 2014/03/10 21:26 OK clematis PING PING OK - Packet loss = 0%, RTA = 0.26 ms [21:27:44] 2014/03/10 21:26 OK clematis SMTP SMTP OK - 0.260 sec. response time [21:27:44] 2014/03/10 21:27 OK clematis SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:28:44] 2014/03/10 21:27 OK ortelius / DISK OK - free space: / 10256 MB (34% inode=91%): [21:28:44] 2014/03/10 21:27 OK ortelius /tmp DISK OK - free space: /tmp 25437 MB (99% inode=99%): [21:28:44] 2014/03/10 21:28 OK ortelius DiskSuite OK - No disk failures detected [21:28:44] 2014/03/10 21:28 OK ortelius Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [21:28:45] 2014/03/10 21:28 OK ortelius Load avg. OK - load average: 2.93, 1.18, 0.45 [21:28:45] 2014/03/10 21:27 OK ortelius PING PING OK - Packet loss = 0%, RTA = 0.11 ms [21:28:45] 2014/03/10 21:27 OK ortelius SMTP SMTP OK - 0.002 sec. response time [21:28:46] 2014/03/10 21:27 OK ortelius SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:28:46] 2014/03/10 21:27 WARN ortelius Sun Grid Engine execd NRPE: Unable to read output [21:28:47] 2014/03/10 21:27 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time [21:40:44] 2014/03/10 21:40 CRIT wolfsbane NTP CRITICAL - Socket timeout after 10 seconds [21:40:44] 2014/03/10 21:40 CRIT wolfsbane PING CRITICAL - Host Unreachable (wolfsbane) [21:42:45] 2014/03/10 21:41 OK wolfsbane /tmp DISK OK - free space: / 10689 MB (35% inode=93%): [21:42:45] 2014/03/10 21:42 OK wolfsbane Load avg. OK - load average: 1.78, 0.70, 0.26 [21:42:45] 2014/03/10 21:42 OK wolfsbane NTP NTP OK: Offset -0.053157 secs [21:42:45] 2014/03/10 21:42 OK wolfsbane PING PING OK - Packet loss = 0%, RTA = 0.29 ms [21:42:45] 2014/03/10 21:42 OK wolfsbane SMTP SMTP OK - 0.569 sec. response time [21:43:45] 2014/03/10 21:42 OK wolfsbane / DISK OK - free space: / 11425 MB (38% inode=93%): [21:43:45] 2014/03/10 21:43 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [21:43:45] 2014/03/10 21:42 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:43:45] 2014/03/10 21:42 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [21:43:45] 2014/03/10 21:42 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.119 second response time [21:48:45] 2014/03/10 21:47 CRIT wolfsbane /tmp Timeout while attempting connection [21:48:45] 2014/03/10 21:48 CRIT wolfsbane Environment IPMI Timeout while attempting connection [21:48:45] 2014/03/10 21:48 CRIT wolfsbane Load avg. Timeout while attempting connection [21:48:45] 2014/03/10 21:48 CRIT wolfsbane NTP CRITICAL - Socket timeout after 10 seconds [21:48:45] 2014/03/10 21:48 CRIT wolfsbane PING CRITICAL - Host Unreachable (wolfsbane) [21:48:46] 2014/03/10 21:47 CRIT wolfsbane Sun Grid Engine execd Connection refused by host [21:49:45] 2014/03/10 21:48 CRIT wolfsbane / Connection refused or timed out [21:49:45] 2014/03/10 21:48 CRIT wolfsbane SMTP No route to host [21:49:45] 2014/03/10 21:48 CRIT wolfsbane SSH No route to host [21:49:45] 2014/03/10 21:48 CRIT wolfsbane toolserver.org HTTP No route to host [21:50:46] 2014/03/10 21:50 OK wolfsbane PING PING OK - Packet loss = 0%, RTA = 0.14 ms [21:50:47] 2014/03/10 21:50 OK wolfsbane SMTP SMTP OK - 0.150 sec. response time [21:51:47] 2014/03/10 21:50 OK wolfsbane / DISK OK - free space: / 11424 MB (38% inode=93%): [21:51:48] 2014/03/10 21:50 OK wolfsbane /tmp DISK OK - free space: / 11424 MB (38% inode=93%): [21:51:48] 2014/03/10 21:51 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [21:51:48] 2014/03/10 21:51 OK wolfsbane Load avg. OK - load average: 1.60, 0.79, 0.30 [21:51:48] 2014/03/10 21:50 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:51:48] 2014/03/10 21:50 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [21:51:48] 2014/03/10 21:50 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.013 second response time