[03:10:01] 2013/06/20 03:03 CRIT yarrow aliasd Connection refused [05:48:10] 2013/06/20 05:47 OK yarrow aliasd TCP OK - 0.003 second response time on port 984 [500 Not found.] [05:55:10] 2013/06/20 05:54 OK ortelius / DISK OK - free space: / 6322 MB (21% inode=92%): [06:09:50] #wikipedia-fr [09:26:19] 2013/06/20 09:19 WARN z-dat-s1-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2171 [09:32:20] 2013/06/20 09:25 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2097 [09:33:20] 2013/06/20 09:26 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2143.000000 [09:51:21] 2013/06/20 09:50 CRIT z-dat-s1-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [09:54:21] 2013/06/20 09:53 OK daphne MySQL slave Uptime: 5407619 Threads: 19 Questions: 538182707 Slow queries: 1025788 Opens: 169258 Flush tables: 1 Open tables: 2093 Queries per second avg: 99.523 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1373 [09:54:21] 2013/06/20 09:53 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1458.000000 [09:55:21] 2013/06/20 09:54 WARN z-dat-s1-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3187 [10:22:22] 2013/06/20 10:21 OK z-dat-s1-b MySQL slave Uptime: 2771832 Threads: 9 Questions: 2655916447 Slow queries: 43908 Opens: 14129 Flush tables: 1 Open tables: 690 Queries per second avg: 958.180 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1757 [12:20:29] 2013/06/20 12:13 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2120 [13:12:30] 2013/06/20 13:11 OK z-dat-s2-b MySQL slave Uptime: 2782115 Threads: 6 Questions: 2756801029 Slow queries: 55960 Opens: 19735772 Flush tables: 1 Open tables: 256 Queries per second avg: 990.901 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1776 [13:14:31] 2013/06/20 13:14 CRIT z-dat-s1-b /sql DISK CRITICAL - free space: /sql 58324 MB (5% inode=99%): [14:37:32] 2013/06/20 14:37 WARN z-dat-s1-b /sql DISK WARNING - free space: /sql 59931 MB (6% inode=99%): [15:35:35] 2013/06/20 15:31 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (111) [15:38:35] 2013/06/20 15:31 CRIT z-dat-s6-a wikidata replag QUERY CRITICAL: Can't connect to MySQL server on 'z-dat-s6-a' (111) [15:39:35] 2013/06/20 15:32 CRIT z-dat-s6-a MySQL slave Can't connect to MySQL server on 'z-dat-s6-a' (111) [15:39:35] 2013/06/20 15:32 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2147 [15:52:36] 2013/06/20 15:52 WARN willow Load avg. WARNING - load average: 19.19, 19.60, 20.00 [15:55:36] 2013/06/20 15:55 CRIT willow Load avg. CRITICAL - load average: 24.70, 20.66, 20.27 [15:59:37] 2013/06/20 15:59 WARN willow Load avg. WARNING - load average: 18.61, 19.64, 19.95 [16:00:38] 2013/06/20 16:00 CRIT willow Load avg. CRITICAL - load average: 24.51, 20.95, 20.38 [16:07:36] 2013/06/20 16:06 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2105.000000 [16:08:36] 2013/06/20 16:07 OK z-dat-s6-a MySQL Uptime: 2157 Threads: 9 Questions: 969 Slow queries: 0 Opens: 102 Flush tables: 1 Open tables: 92 Queries per second avg: 0.449 [16:08:36] 2013/06/20 16:07 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2160 [16:08:36] hello [16:08:43] hardware maintenance tonight [16:09:06] our local dc contact will arrive in about 1 hour and we will fix some stuff [16:09:27] some small outages might be possible but nothing serious [16:09:36] 2013/06/20 16:09 WARN willow Load avg. WARNING - load average: 18.83, 19.55, 19.97 [16:10:36] 2013/06/20 16:10 CRIT willow Load avg. CRITICAL - load average: 22.27, 20.28, 20.20 [16:12:36] 2013/06/20 16:11 CRIT z-dat-s7-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3649 [16:13:36] 2013/06/20 16:06 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2125.000000 [16:14:36] 2013/06/20 16:13 OK z-dat-s6-a MySQL slave Uptime: 2509 Threads: 8 Questions: 44102 Slow queries: 28 Opens: 236 Flush tables: 1 Open tables: 225 Queries per second avg: 17.577 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1734 [16:32:39] 2013/06/20 16:31 CRIT z-dat-s6-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3604.000000 [16:33:39] 2013/06/20 16:32 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3495 [16:40:40] 2013/06/20 16:39 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1697.000000 [16:43:40] 2013/06/20 16:42 OK z-dat-s7-a MySQL slave Uptime: 848845 Threads: 5 Questions: 429899317 Slow queries: 21568 Opens: 799440 Flush tables: 1 Open tables: 3781 Queries per second avg: 506.452 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1754 [16:51:40] 2013/06/20 16:50 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3583.000000 [17:18:37] hemlock offline [17:18:40] gets repair [17:18:42] 2013/06/20 17:18 CRIT hemlock / Connection refused or timed out [17:18:42] 2013/06/20 17:18 CRIT hemlock /aux0 Connection refused or timed out [17:18:42] 2013/06/20 17:17 CRIT hemlock /home Connection refused or timed out [17:18:42] 2013/06/20 17:17 CRIT hemlock Environment IPMI Connection refused or timed out [17:18:43] 2013/06/20 17:17 CRIT hemlock Load avg. Connection refused or timed out [17:18:43] 2013/06/20 17:17 CRIT hemlock NTP CRITICAL - Socket timeout after 10 seconds [17:18:43] 2013/06/20 17:17 CRIT hemlock PING CRITICAL - Host Unreachable (hemlock) [17:18:44] 2013/06/20 17:17 CRIT hemlock SMTP CRITICAL - Socket timeout after 10 seconds [17:18:44] 2013/06/20 17:17 CRIT hemlock SSH No route to host [17:19:42] 2013/06/20 17:18 CRIT hemlock /tmp Connection refused or timed out [17:19:42] 2013/06/20 17:18 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1792.000000 [17:20:42] 2013/06/20 17:19 CRIT hemlock.mgmt PING PING CRITICAL - Packet loss = 100% [17:20:43] 2013/06/20 17:19 CRIT hemlock.mgmt SSH CRITICAL - Socket timeout after 10 seconds [17:23:43] 2013/06/20 17:23 CRIT hemlock CRITICAL - Host Unreachable (hemlock) [17:25:43] 2013/06/20 17:25 WARN willow Load avg. WARNING - load average: 19.49, 19.56, 19.99 [17:26:43] 2013/06/20 17:26 CRIT willow Load avg. CRITICAL - load average: 20.17, 19.73, 20.02 [17:27:43] 2013/06/20 17:27 WARN willow Load avg. WARNING - load average: 19.53, 19.68, 19.99 [17:28:43] 2013/06/20 17:27 OK hemlock.mgmt PING PING OK - Packet loss = 0%, RTA = 0.47 ms [17:28:43] 2013/06/20 17:27 OK hemlock.mgmt SSH SSH OK - OpenSSH_3.8.1p1 (protocol 2.0) [17:30:44] 2013/06/20 17:30 CRIT willow Load avg. CRITICAL - load average: 26.41, 21.25, 20.46 [17:31:44] 2013/06/20 17:31 OK hemlock PING OK - Packet loss = 0%, RTA = 0.10 ms [17:32:44] 2013/06/20 17:31 OK hemlock PING PING OK - Packet loss = 0%, RTA = 0.24 ms [17:53:21] ok hemlock back with working san [17:53:33] seems we can move the file server back [17:57:47] 2013/06/20 17:57 WARN willow Load avg. WARNING - load average: 19.26, 19.62, 19.94 [18:00:46] 2013/06/20 18:00 CRIT willow Load avg. CRITICAL - load average: 29.77, 22.09, 20.72 [18:10:46] 2013/06/20 18:10 WARN willow Load avg. WARNING - load average: 14.56, 18.68, 19.84 [18:19:09] wolfsbane gets maintenance [18:19:13] ill take it off [18:19:46] 2013/06/20 18:19 OK willow Load avg. OK - load average: 8.60, 10.52, 14.93 [18:21:46] 2013/06/20 18:20 CRIT wolfsbane / Timeout while attempting connection [18:21:46] 2013/06/20 18:21 CRIT wolfsbane /tmp Connection refused or timed out [18:21:46] 2013/06/20 18:20 CRIT wolfsbane Environment IPMI Connection refused or timed out [18:21:46] 2013/06/20 18:21 CRIT wolfsbane Load avg. Connection refused or timed out [18:21:46] 2013/06/20 18:20 CRIT wolfsbane NTP CRITICAL - Socket timeout after 10 seconds [18:21:47] 2013/06/20 18:20 CRIT wolfsbane PING CRITICAL - Host Unreachable (wolfsbane) [18:21:47] 2013/06/20 18:20 CRIT wolfsbane Sun Grid Engine execd Connection refused or timed out [18:21:48] 2013/06/20 18:20 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [18:22:46] 2013/06/20 18:21 CRIT wolfsbane SMTP No route to host [18:22:46] 2013/06/20 18:21 CRIT wolfsbane SSH No route to host [18:27:47] 2013/06/20 18:27 CRIT wolfsbane CRITICAL - Host Unreachable (wolfsbane) [18:36:46] 2013/06/20 18:36 OK daphne RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:37:46] 2013/06/20 18:37 CRIT rosemary.mgmt PING PING CRITICAL - Packet loss = 100% [18:38:46] 2013/06/20 18:38 OK wolfsbane PING OK - Packet loss = 0%, RTA = 0.14 ms [18:38:46] 2013/06/20 18:38 OK rosemary.mgmt PING PING OK - Packet loss = 0%, RTA = 0.70 ms [18:38:46] 2013/06/20 18:38 WARN willow.mgmt PING PING WARNING - Packet loss = 50%, RTA = 0.50 ms [18:38:46] 2013/06/20 18:38 OK wolfsbane Load avg. OK - load average: 5.47, 1.86, 0.68 [18:38:46] 2013/06/20 18:38 OK wolfsbane PING PING OK - Packet loss = 0%, RTA = 0.11 ms [18:38:47] 2013/06/20 18:38 OK wolfsbane SMTP SMTP OK - 0.581 sec. response time [18:39:46] 2013/06/20 18:39 OK willow.mgmt PING PING OK - Packet loss = 0%, RTA = 0.29 ms [18:39:46] 2013/06/20 18:38 OK wolfsbane / DISK OK - free space: / 14160 MB (47% inode=93%): [18:39:46] 2013/06/20 18:39 OK wolfsbane /tmp DISK OK - free space: / 14092 MB (47% inode=93%): [18:39:46] 2013/06/20 18:38 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [18:39:46] 2013/06/20 18:38 OK wolfsbane NTP NTP OK: Offset -0.043023 secs [18:39:47] 2013/06/20 18:38 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:39:47] 2013/06/20 18:38 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [18:39:48] 2013/06/20 18:38 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.008 second response time [19:34:57] is apache down? [23:00:09] If it is possible, could somebody get a crontab for me?