[03:08:19] 2013/07/07 03:07 WARN wolfsbane / DISK WARNING - free space: / 3381 MB (11% inode=92%): [03:08:19] 2013/07/07 03:07 WARN wolfsbane /tmp DISK WARNING - free space: / 3374 MB (11% inode=92%): [03:23:21] 2013/07/07 03:16 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2130.000000 [03:39:22] 2013/07/07 03:32 CRIT nightshade aliasd Connection refused [03:39:22] 2013/07/07 03:38 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1733.000000 [03:54:23] 2013/07/07 03:53 CRIT wolfsbane / DISK CRITICAL - free space: / 3294 MB (10% inode=92%): [03:56:23] 2013/07/07 03:55 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3294 MB (10% inode=92%): [03:57:23] 2013/07/07 03:56 WARN wolfsbane / DISK WARNING - free space: / 3296 MB (11% inode=92%): [03:58:23] 2013/07/07 03:57 CRIT wolfsbane / DISK CRITICAL - free space: / 3293 MB (10% inode=92%): [03:58:24] 2013/07/07 03:57 WARN wolfsbane /tmp DISK WARNING - free space: / 3301 MB (11% inode=92%): [03:59:24] 2013/07/07 03:58 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3285 MB (10% inode=92%): [04:08:24] 2013/07/07 04:07 WARN wolfsbane / DISK WARNING - free space: / 4568 MB (15% inode=92%): [04:08:25] 2013/07/07 04:07 WARN wolfsbane /tmp DISK WARNING - free space: / 4603 MB (15% inode=92%): [09:50:46] 2013/07/07 09:49 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:45] 2013/07/07 09:51 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 66040 MB (10% inode=99%): [09:54:45] 2013/07/07 09:51 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [09:54:45] 2013/07/07 09:51 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [10:10:46] 2013/07/07 10:10 OK z-dat-s3-a MySQL Uptime: 1181 Threads: 12 Questions: 468 Slow queries: 0 Opens: 193 Flush tables: 1 Open tables: 184 Queries per second avg: 0.396 [10:10:46] 2013/07/07 10:10 OK z-dat-s3-a MySQL slave Uptime: 1181 Threads: 10 Questions: 470 Slow queries: 0 Opens: 193 Flush tables: 1 Open tables: 184 Queries per second avg: 0.397 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1185 [10:28:47] 2013/07/07 10:25 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1835 [10:29:47] 2013/07/07 10:29 OK z-dat-s3-a MySQL slave Uptime: 2322 Threads: 21 Questions: 352819 Slow queries: 50 Opens: 10839 Flush tables: 1 Open tables: 7444 Queries per second avg: 151.946 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1765 [10:35:47] 2013/07/07 10:32 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1973 [10:41:48] 2013/07/07 10:41 OK z-dat-s3-a MySQL slave Uptime: 3042 Threads: 20 Questions: 1385891 Slow queries: 98 Opens: 18724 Flush tables: 1 Open tables: 10026 Queries per second avg: 455.585 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1745 [10:51:49] 2013/07/07 10:47 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [10:51:49] 2013/07/07 10:47 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (110) [10:51:49] 2013/07/07 10:47 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [10:53:48] 2013/07/07 10:46 CRIT hyacinth Environment IPMI CHECK_NRPE: Socket timeout after 30 seconds. [10:53:48] 2013/07/07 10:47 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [10:53:48] 2013/07/07 10:47 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [10:53:48] 2013/07/07 10:46 CRIT z-dat-s7-a / CHECK_NRPE: Socket timeout after 30 seconds. [10:53:48] 2013/07/07 10:46 CRIT z-dat-s7-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [10:54:48] 2013/07/07 10:53 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [10:54:48] 2013/07/07 10:54 OK hyacinth SMTP SMTP OK - 0.003 sec. response time [10:54:48] 2013/07/07 10:54 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:54:48] 2013/07/07 10:53 OK z-dat-s3-a MySQL Uptime: 3781 Threads: 17 Questions: 1829661 Slow queries: 151 Opens: 19377 Flush tables: 1 Open tables: 10677 Queries per second avg: 483.909 [10:54:48] 2013/07/07 10:53 OK z-dat-s3-a MySQL slave Uptime: 3781 Threads: 17 Questions: 1829660 Slow queries: 151 Opens: 19377 Flush tables: 1 Open tables: 10677 Queries per second avg: 483.909 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1352 [10:54:49] 2013/07/07 10:53 OK z-dat-s6-a MySQL Uptime: 1338233 Threads: 7 Questions: 868302466 Slow queries: 52103 Opens: 316950 Flush tables: 1 Open tables: 3557 Queries per second avg: 648.842 [10:54:49] 2013/07/07 10:53 OK z-dat-s7-a / DISK OK - free space: / 6339 MB (21% inode=84%): [10:54:50] 2013/07/07 10:53 OK z-dat-s7-a /sql DISK OK - free space: /sql 138053 MB (19% inode=99%): [11:20:50] 2013/07/07 11:19 CRIT wolfsbane / DISK CRITICAL - free space: / 3267 MB (10% inode=92%): [11:20:51] 2013/07/07 11:20 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3270 MB (10% inode=92%): [11:22:51] 2013/07/07 11:21 WARN wolfsbane / DISK WARNING - free space: / 3311 MB (11% inode=92%): [11:22:51] 2013/07/07 11:22 WARN wolfsbane /tmp DISK WARNING - free space: / 3304 MB (11% inode=92%): [11:26:51] 2013/07/07 11:26 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3287 MB (10% inode=92%): [11:27:51] 2013/07/07 11:26 CRIT wolfsbane / DISK CRITICAL - free space: / 3277 MB (10% inode=92%): [12:14:07] hello all [14:50:03] 2013/07/07 14:43 WARN ptolemy Load avg. WARNING - load average: 19.96, 18.35, 12.71 [14:55:04] 2013/07/07 14:54 OK ptolemy Load avg. OK - load average: 11.92, 14.70, 12.73 [15:05:04] 2013/07/07 15:01 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1997 [15:08:05] 2013/07/07 15:00 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1941.000000 [15:11:05] 2013/07/07 15:10 OK z-dat-s5-b MySQL slave Uptime: 1216831 Threads: 18 Questions: 1604165131 Slow queries: 6768 Opens: 590567 Flush tables: 1 Open tables: 256 Queries per second avg: 1318.313 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [15:11:05] 2013/07/07 15:09 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1779.000000 [19:05:19] 2013/07/07 19:04 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.607 second response time [19:17:20] 2013/07/07 19:16 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.784 second response time [19:18:20] 2013/07/07 19:17 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.600 second response time [19:19:21] 2013/07/07 19:18 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.823 second response time [19:20:21] 2013/07/07 19:19 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 6.039 second response time [19:24:21] 2013/07/07 19:23 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.323 second response time [22:04:26] 2013/07/07 22:02 CRIT wolfsbane Sun Grid Engine execd CHECK_NRPE: Socket timeout after 30 seconds. [22:06:26] 2013/07/07 22:02 CRIT wolfsbane Environment IPMI Timeout while attempting connection [22:07:26] 2013/07/07 22:01 CRIT wolfsbane Load avg. Timeout while attempting connection [22:09:26] 2013/07/07 22:02 CRIT wolfsbane SSH CRITICAL - Socket timeout after 10 seconds [22:09:26] 2013/07/07 22:02 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [22:20:28] 2013/07/07 22:13 CRIT wolfsbane NTP CRITICAL - Socket timeout after 10 seconds [22:43:31] 2013/07/07 22:36 CRIT wolfsbane SMTP CRITICAL - Socket timeout after 10 seconds