[00:27:26] 2014/01/31 00:20 WARN yarrow Load avg. WARNING - load average: 24.71, 19.06, 10.64 [00:31:27] 2014/01/31 00:30 CRIT yarrow Load avg. CRITICAL - load average: 30.59, 24.80, 14.86 [02:18:32] 2014/01/31 02:11 CRIT rosemary MySQL Can't connect to MySQL server on 'rosemary' (111) [02:29:32] 2014/01/31 02:28 OK rosemary MySQL Uptime: 1016 Threads: 8 Questions: 6078 Slow queries: 2 Opens: 100 Flush tables: 1 Open tables: 87 Queries per second avg: 5.982 [03:08:34] 2014/01/31 03:07 WARN wolfsbane / DISK WARNING - free space: / 3706 MB (12% inode=93%): [03:08:34] 2014/01/31 03:07 WARN wolfsbane /tmp DISK WARNING - free space: / 3706 MB (12% inode=93%): [03:50:37] 2014/01/31 03:43 WARN nightshade Load avg. WARNING - load average: 24.51, 18.96, 11.13 [03:54:37] 2014/01/31 03:53 CRIT nightshade Load avg. CRITICAL - load average: 31.25, 25.09, 15.37 [04:01:38] 2014/01/31 04:00 CRIT wolfsbane / DISK CRITICAL - free space: / 3278 MB (10% inode=93%): [04:01:38] 2014/01/31 04:00 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3278 MB (10% inode=93%): [04:04:38] 2014/01/31 04:03 OK wolfsbane / DISK OK - free space: / 7574 MB (25% inode=93%): [04:04:38] 2014/01/31 04:03 OK wolfsbane /tmp DISK OK - free space: / 7574 MB (25% inode=93%): [04:04:38] 2014/01/31 03:57 CRIT wolfsbane toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [04:12:38] 2014/01/31 04:11 WARN wolfsbane toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.566 second response time [04:13:38] 2014/01/31 04:12 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.012 second response time [04:23:38] 2014/01/31 04:22 WARN nightshade Load avg. WARNING - load average: 3.65, 8.30, 19.90 [04:29:38] 2014/01/31 04:28 OK nightshade Load avg. OK - load average: 3.75, 4.96, 14.63 [04:41:39] 2014/01/31 04:40 WARN cassia /sql DISK WARNING - free space: /sql 72121 MB (6% inode=99%): [04:42:39] 2014/01/31 04:41 CRIT cassia /sql DISK CRITICAL - free space: /sql 69168 MB (5% inode=99%): [04:56:40] 2014/01/31 04:55 WARN yarrow Load avg. WARNING - load average: 0.00, 0.04, 18.97 [05:00:40] 2014/01/31 04:59 OK yarrow Load avg. OK - load average: 0.00, 0.01, 14.64 [05:26:50] [[Special:Log/newusers]] create 10 * Lon889 * (New user account) [06:35:46] 2014/01/31 06:29 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2161.000000 [06:36:46] 2014/01/31 06:29 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2216 [06:52:47] 2014/01/31 06:52 WARN cassia /sql DISK WARNING - free space: /sql 71208 MB (6% inode=99%): [06:53:47] 2014/01/31 06:53 CRIT cassia /sql DISK CRITICAL - free space: /sql 70207 MB (5% inode=99%): [06:59:48] 2014/01/31 06:59 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3601.000000 [07:00:48] 2014/01/31 07:00 WARN cassia /sql DISK WARNING - free space: /sql 71024 MB (6% inode=99%): [07:00:49] 2014/01/31 06:59 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3656 [07:01:50] 2014/01/31 07:01 CRIT cassia /sql DISK CRITICAL - free space: /sql 70186 MB (5% inode=99%): [07:46:52] 2014/01/31 07:40 WARN wolfsbane / DISK WARNING - free space: / 6262 MB (20% inode=93%): [07:46:52] 2014/01/31 07:40 WARN wolfsbane /tmp DISK WARNING - free space: / 6262 MB (20% inode=93%): [07:48:51] 2014/01/31 07:48 OK wolfsbane / DISK OK - free space: / 6315 MB (21% inode=93%): [07:48:51] 2014/01/31 07:48 OK wolfsbane /tmp DISK OK - free space: / 6314 MB (21% inode=93%): [07:54:01] [[Special:Log/newusers]] create 10 * Rickypojan * (New user account) [07:57:51] 2014/01/31 07:51 WARN wolfsbane / DISK WARNING - free space: / 6220 MB (20% inode=93%): [07:57:51] 2014/01/31 07:51 WARN wolfsbane /tmp DISK WARNING - free space: / 6220 MB (20% inode=93%): [08:06:08] [[Special:Log/newusers]] create 10 * Tenisha985 * (New user account) [08:32:53] 2014/01/31 08:32 WARN cassia /sql DISK WARNING - free space: /sql 72365 MB (6% inode=99%): [08:33:53] 2014/01/31 08:33 CRIT cassia /sql DISK CRITICAL - free space: /sql 70052 MB (5% inode=99%): [09:45:57] 2014/01/31 09:45 WARN cassia /sql DISK WARNING - free space: /sql 78903 MB (6% inode=99%): [09:46:57] 2014/01/31 09:46 CRIT cassia /sql DISK CRITICAL - free space: /sql 69691 MB (5% inode=99%): [09:53:58] 2014/01/31 09:52 OK nightshade aliasd TCP OK - 0.001 second response time on port 984 [500 Not found.] [10:02:58] 2014/01/31 10:01 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3300 [10:02:59] 2014/01/31 10:02 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3174.000000 [10:04:59] 2014/01/31 10:04 OK wolfsbane / DISK OK - free space: / 9962 MB (33% inode=93%): [10:04:59] 2014/01/31 10:04 OK wolfsbane /tmp DISK OK - free space: / 9962 MB (33% inode=93%): [10:04:59] 2014/01/31 10:03 OK z-dat-s5-b MySQL slave Uptime: 605113 Threads: 6 Questions: 218057432 Slow queries: 262 Opens: 331300 Flush tables: 2 Open tables: 256 Queries per second avg: 360.358 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1325 [10:04:59] 2014/01/31 10:04 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1323.000000 [10:06:58] 2014/01/31 10:06 WARN cassia /sql DISK WARNING - free space: /sql 72247 MB (6% inode=99%): [10:07:58] 2014/01/31 10:07 CRIT cassia /sql DISK CRITICAL - free space: /sql 68270 MB (5% inode=99%): [11:53:05] 2014/01/31 11:51 OK z-dat-s4-a MySQL Uptime: 48 Threads: 4 Questions: 5 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 0.104 [11:53:05] 2014/01/31 11:51 WARN z-dat-s4-a MySQL slave No slaves defined [12:14:06] 2014/01/31 12:13 OK z-dat-s1-b /sql DISK OK - free space: /sql 978883 MB (99% inode=99%): [12:14:06] 2014/01/31 12:13 CRIT z-dat-s1-b MySQL slave Can't connect to MySQL server on 'z-dat-s1-b' (111) [12:19:06] 2014/01/31 12:12 CRIT z-dat-s1-b MySQL Can't connect to MySQL server on 'z-dat-s1-b' (111) [13:47:09] 2014/01/31 13:40 WARN wolfsbane / DISK WARNING - free space: / 5970 MB (19% inode=93%): [13:47:09] 2014/01/31 13:40 WARN wolfsbane /tmp DISK WARNING - free space: / 5970 MB (19% inode=93%): [14:09:10] 2014/01/31 14:01 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [15:08:12] 2014/01/31 15:07 OK wolfsbane / DISK OK - free space: / 6988 MB (23% inode=93%): [15:08:12] 2014/01/31 15:07 OK wolfsbane /tmp DISK OK - free space: / 6988 MB (23% inode=93%): [15:24:14] 2014/01/31 15:23 WARN cassia /sql DISK WARNING - free space: /sql 73478 MB (6% inode=99%): [15:26:14] 2014/01/31 15:25 CRIT cassia /sql DISK CRITICAL - free space: /sql 70498 MB (5% inode=99%): [15:37:14] 2014/01/31 15:36 WARN cassia /sql DISK WARNING - free space: /sql 71023 MB (6% inode=99%): [15:39:15] 2014/01/31 15:38 CRIT cassia /sql DISK CRITICAL - free space: /sql 70084 MB (5% inode=99%): [16:01:17] 2014/01/31 15:54 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [16:02:17] 2014/01/31 16:01 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [16:04:17] 2014/01/31 15:57 WARN wolfsbane / DISK WARNING - free space: / 6139 MB (20% inode=93%): [16:04:18] 2014/01/31 15:57 WARN wolfsbane /tmp DISK WARNING - free space: / 6140 MB (20% inode=93%): [16:55:22] 2014/01/31 16:54 WARN cassia /sql DISK WARNING - free space: /sql 71998 MB (6% inode=99%): [16:56:22] 2014/01/31 16:55 CRIT cassia /sql DISK CRITICAL - free space: /sql 69895 MB (5% inode=99%): [17:41:25] 2014/01/31 17:34 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [17:48:26] 2014/01/31 17:45 CRIT nightshade Load avg. CRITICAL - load average: 35.23, 25.82, 13.67 [18:07:27] 2014/01/31 18:06 CRIT wolfsbane / DISK CRITICAL - free space: / 3280 MB (10% inode=93%): [18:07:27] 2014/01/31 18:06 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3280 MB (10% inode=93%): [18:18:12] [[Special:Log/newusers]] create 10 * Fausto398 * (New user account) [19:08:28] 2014/01/31 19:07 WARN wolfsbane / DISK WARNING - free space: / 3945 MB (13% inode=93%): [19:08:28] 2014/01/31 19:07 WARN wolfsbane /tmp DISK WARNING - free space: / 3946 MB (13% inode=93%): [19:56:33] 2014/01/31 19:55 CRIT wolfsbane / DISK CRITICAL - free space: / 3145 MB (10% inode=93%): [19:56:33] 2014/01/31 19:55 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3144 MB (10% inode=93%): [20:25:34] 2014/01/31 20:24 WARN wolfsbane / DISK WARNING - free space: / 5304 MB (17% inode=93%): [20:25:34] 2014/01/31 20:24 WARN wolfsbane /tmp DISK WARNING - free space: / 5304 MB (17% inode=93%): [20:27:34] 2014/01/31 20:26 OK wolfsbane / DISK OK - free space: / 6449 MB (21% inode=93%): [20:27:34] 2014/01/31 20:26 OK wolfsbane /tmp DISK OK - free space: / 6449 MB (21% inode=93%): [20:34:35] 2014/01/31 20:28 WARN willow / DISK WARNING - free space: / 21701 MB (20% inode=98%): [20:34:35] 2014/01/31 20:27 WARN willow /tmp DISK WARNING - free space: / 21701 MB (20% inode=98%): [20:38:34] 2014/01/31 20:38 OK willow / DISK OK - free space: / 25443 MB (24% inode=99%): [20:38:34] 2014/01/31 20:37 OK willow /tmp DISK OK - free space: / 25443 MB (24% inode=99%): [20:48:35] 2014/01/31 20:41 WARN wolfsbane / DISK WARNING - free space: / 6241 MB (20% inode=93%): [20:48:35] 2014/01/31 20:41 WARN wolfsbane /tmp DISK WARNING - free space: / 6241 MB (20% inode=93%): [21:00:36] 2014/01/31 20:59 OK wolfsbane / DISK OK - free space: / 9524 MB (31% inode=93%): [21:00:36] 2014/01/31 20:59 OK wolfsbane /tmp DISK OK - free space: / 9524 MB (31% inode=93%): [21:03:36] 2014/01/31 21:02 CRIT nightshade Environment IPMI Timeout while attempting connection [21:03:36] 2014/01/31 21:02 CRIT nightshade Sensors Timeout while attempting connection [21:04:36] 2014/01/31 21:03 CRIT nightshade / Connection refused or timed out [21:04:36] 2014/01/31 21:03 CRIT nightshade /tmp Connection refused or timed out [21:04:36] 2014/01/31 21:03 CRIT nightshade /var Connection refused or timed out [21:04:36] 2014/01/31 21:03 CRIT nightshade /var/tmp Connection refused or timed out [21:04:36] 2014/01/31 21:03 CRIT nightshade NTP CRITICAL - Socket timeout after 10 seconds [21:04:37] 2014/01/31 21:03 CRIT nightshade PING CRITICAL - Host Unreachable (nightshade) [21:04:37] 2014/01/31 21:03 CRIT nightshade SMTP CRITICAL - Socket timeout after 10 seconds [21:04:38] 2014/01/31 21:03 CRIT nightshade SSH CRITICAL - Socket timeout after 10 seconds [21:04:38] 2014/01/31 21:03 CRIT nightshade Sun Grid Engine execd Connection refused or timed out [21:06:37] 2014/01/31 21:05 OK nightshade / DISK OK - free space: / 1592 MB (89% inode=94%): [21:06:37] 2014/01/31 21:05 OK nightshade /tmp DISK OK - free space: /tmp 4317 MB (96% inode=99%): [21:06:37] 2014/01/31 21:05 OK nightshade /var DISK OK - free space: /var 9381 MB (70% inode=47%): [21:06:37] 2014/01/31 21:05 OK nightshade /var/tmp DISK OK - free space: /var/tmp 872 MB (98% inode=99%): [21:06:37] 2014/01/31 21:05 OK nightshade Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [21:06:37] 2014/01/31 21:05 OK nightshade Load avg. OK - load average: 1.30, 0.29, 0.09 [21:06:38] 2014/01/31 21:05 OK nightshade PING PING OK - Packet loss = 0%, RTA = 0.35 ms [21:06:38] 2014/01/31 21:05 OK nightshade SMTP SMTP OK - 0.551 sec. response time [21:06:39] 2014/01/31 21:05 OK nightshade SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [21:06:39] 2014/01/31 21:05 OK nightshade Sensors sensor ok [21:06:40] 2014/01/31 21:05 OK nightshade aliasd TCP OK - 0.002 second response time on port 984 [500 Not found.] [21:12:20] omg, who restarted nightshade? [21:13:36] 2014/01/31 21:06 WARN nightshade NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 0.062274 secs [21:13:36] 2014/01/31 21:03 CRIT nightshade Sun Grid Engine execd CRITICAL: execd not communicating [21:19:08] Danny_B: Not me; but http://munin.toolserver.org/Login/nightshade/index.html#processes looks interesting; lots of uninterruptible processes? [21:22:36] 2014/01/31 21:21 CRIT nightshade NTP NTP CRITICAL: Server not synchronized, Offset unknown [21:24:37] 2014/01/31 21:23 WARN nightshade NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 0.000589 secs [21:33:38] 2014/01/31 21:32 OK nightshade Sun Grid Engine execd Host and Queues Ok [21:33:38] 2014/01/31 21:26 CRIT yucca SSH CRITICAL - Socket timeout after 10 seconds [21:35:38] 2014/01/31 21:34 OK yucca SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [21:37:39] 2014/01/31 21:36 OK nightshade NTP NTP OK: Offset 0.000713 secs [22:40:41] 2014/01/31 22:33 CRIT nightshade aliasd CRITICAL - Socket timeout after 10 seconds [22:45:42] 2014/01/31 22:44 CRIT nightshade Load avg. CRITICAL - load average: 30.63, 21.46, 10.57 [23:01:11] [[Special:Log/newusers]] create 10 * Hassen.Houssein * (New user account) [23:44:02] [[Special:Log/newusers]] create 10 * Mitsuko877 * (New user account)