[00:10:08] 2013/08/11 00:03 CRIT nightshade aliasd Connection refused [02:56:19] 2013/08/11 02:52 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1987 [02:59:19] 2013/08/11 02:52 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2088.000000 [03:12:20] 2013/08/11 03:05 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1891.000000 [03:35:19] 2013/08/11 03:34 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3608 [03:35:19] 2013/08/11 03:34 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3608.000000 [03:44:20] 2013/08/11 03:43 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1715.000000 [03:53:22] 2013/08/11 03:52 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3537 [03:53:22] 2013/08/11 03:52 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3538.000000 [04:07:22] 2013/08/11 04:06 OK z-dat-s5-b MySQL slave Uptime: 613465 Threads: 10 Questions: 552648120 Slow queries: 2733 Opens: 281486 Flush tables: 1 Open tables: 256 Queries per second avg: 900.863 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1690 [04:07:23] 2013/08/11 04:06 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1690.000000 [06:11:28] 2013/08/11 06:10 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:12:28] 2013/08/11 06:11 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63361 MB (10% inode=99%): [06:14:29] 2013/08/11 06:10 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [06:14:29] 2013/08/11 06:10 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [06:28:30] 2013/08/11 06:27 OK z-dat-s3-a MySQL Uptime: 1027 Threads: 11 Questions: 59 Slow queries: 0 Opens: 27 Flush tables: 1 Open tables: 18 Queries per second avg: 0.57 [06:28:30] 2013/08/11 06:27 OK z-dat-s3-a MySQL slave Uptime: 1027 Threads: 10 Questions: 60 Slow queries: 0 Opens: 27 Flush tables: 1 Open tables: 18 Queries per second avg: 0.58 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [06:38:04] Silke Meyer * Re: [Toolserver-l] TS and/vs. Labs [06:44:30] 2013/08/11 06:40 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1945 [06:51:30] 2013/08/11 06:50 OK z-dat-s3-a MySQL slave Uptime: 2407 Threads: 18 Questions: 1040408 Slow queries: 201 Opens: 10867 Flush tables: 1 Open tables: 10751 Queries per second avg: 432.242 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1787 [08:16:05] [[Special:Log/newusers]] create 10 * Lavette387 * (New user account) [08:42:19] [[Special:Log/newusers]] create 10 * Britany858 * (New user account) [08:56:36] 2013/08/11 08:54 OK z-dat-s1-b MySQL Uptime: 463 Threads: 5 Questions: 188 Slow queries: 0 Opens: 40 Flush tables: 1 Open tables: 32 Queries per second avg: 0.406 [10:53:42] 2013/08/11 10:49 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (111) [10:54:42] 2013/08/11 10:50 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [10:54:42] 2013/08/11 10:50 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [10:56:42] 2013/08/11 10:50 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (111) [10:57:42] 2013/08/11 10:56 OK z-dat-s6-a MySQL Uptime: 409 Threads: 2 Questions: 9 Slow queries: 0 Opens: 20 Flush tables: 1 Open tables: 13 Queries per second avg: 0.22 [10:57:43] 2013/08/11 10:50 CRIT z-dat-s6-a MySQL slave Can't connect to MySQL server on 'z-dat-s6-a' (111) [10:57:43] 2013/08/11 10:50 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (111) [10:58:42] 2013/08/11 10:57 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1846 [10:59:43] 2013/08/11 10:58 OK z-dat-s6-a MySQL slave Uptime: 487 Threads: 4 Questions: 11978 Slow queries: 32 Opens: 180 Flush tables: 1 Open tables: 172 Queries per second avg: 24.595 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1662 [11:04:42] 2013/08/11 11:04 OK z-dat-s7-a MySQL Uptime: 838 Threads: 1 Questions: 430 Slow queries: 0 Opens: 61 Flush tables: 1 Open tables: 53 Queries per second avg: 0.513 [11:05:42] 2013/08/11 11:04 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2305 [11:07:42] 2013/08/11 11:06 OK z-dat-s3-a MySQL Uptime: 1007 Threads: 14 Questions: 30937 Slow queries: 0 Opens: 227 Flush tables: 1 Open tables: 196 Queries per second avg: 30.721 [11:07:42] 2013/08/11 11:06 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2476 [11:08:42] 2013/08/11 11:07 OK z-dat-s7-a MySQL slave Uptime: 1029 Threads: 5 Questions: 36930 Slow queries: 4 Opens: 448 Flush tables: 1 Open tables: 435 Queries per second avg: 35.889 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1617 [11:26:43] 2013/08/11 11:26 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 56000 MB (5% inode=99%): [11:29:44] 2013/08/11 11:29 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 61428 MB (6% inode=99%): [11:32:44] 2013/08/11 11:31 OK z-dat-s3-a MySQL slave Uptime: 2507 Threads: 18 Questions: 1285785 Slow queries: 76 Opens: 20716 Flush tables: 1 Open tables: 11889 Queries per second avg: 512.877 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1624 [12:17:47] 2013/08/11 12:11 WARN willow /tmp DISK WARNING - free space: / 20272 MB (19% inode=98%): [12:18:47] 2013/08/11 12:11 WARN willow / DISK WARNING - free space: / 20272 MB (19% inode=98%): [12:28:47] 2013/08/11 12:22 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.574 second response time [12:36:47] 2013/08/11 12:36 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.815 second response time [12:37:47] 2013/08/11 12:37 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.042 second response time [12:39:07] [[Special:Log/newusers]] create 10 * Randalrinm * (New user account) [12:39:48] 2013/08/11 12:39 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.570 second response time [12:39:49] 2013/08/11 12:39 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 55853 MB (5% inode=99%): [12:40:49] 2013/08/11 12:40 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.285 second response time [12:46:48] 2013/08/11 12:46 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.170 second response time [12:49:49] 2013/08/11 12:49 WARN z-dat-s5-b /sql DISK WARNING - free space: /sql 58192 MB (6% inode=99%): [13:01:48] 2013/08/11 13:01 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.638 second response time [13:02:48] 2013/08/11 13:02 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.248 second response time [13:02:49] 2013/08/11 13:02 OK willow /tmp DISK OK - free space: / 49830 MB (47% inode=99%): [13:03:48] 2013/08/11 13:02 OK willow / DISK OK - free space: / 49830 MB (47% inode=99%): [16:21:40] [[Special:Log/newusers]] create 10 * Kirk718 * (New user account) [18:13:10] 2013/08/11 18:08 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [18:13:10] 2013/08/11 18:08 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [18:22:11] 2013/08/11 18:15 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2185 [18:26:12] 2013/08/11 18:19 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2141 [18:27:12] 2013/08/11 18:23 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1931 [18:41:13] 2013/08/11 18:39 OK z-dat-s3-a MySQL Uptime: 1844 Threads: 45 Questions: 568 Slow queries: 2 Opens: 177 Flush tables: 1 Open tables: 155 Queries per second avg: 0.308 [18:41:14] 2013/08/11 18:39 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3418 [18:45:14] 2013/08/11 18:43 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3646 [18:47:14] 2013/08/11 18:46 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3626 [18:56:14] 2013/08/11 18:55 CRIT z-dat-s7-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [19:08:14] 2013/08/11 19:07 CRIT z-dat-s4-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3606 [19:48:16] 2013/08/11 19:46 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [19:51:17] 2013/08/11 19:46 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [19:51:17] 2013/08/11 19:46 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [19:51:17] 2013/08/11 19:46 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [19:53:16] 2013/08/11 19:46 CRIT hyacinth Environment IPMI CHECK_NRPE: Socket timeout after 30 seconds. [19:53:16] 2013/08/11 19:46 CRIT hyacinth RAID CHECK_NRPE: Socket timeout after 30 seconds. [19:53:16] 2013/08/11 19:46 CRIT hyacinth SMTP CRITICAL - Socket timeout after 10 seconds [19:53:16] 2013/08/11 19:46 CRIT z-dat-s3-a /tmp Timeout while attempting connection [19:53:16] 2013/08/11 19:46 CRIT z-dat-s3-a Load avg. Timeout while attempting connection [19:53:16] 2013/08/11 19:46 CRIT z-dat-s4-a Load avg. CHECK_NRPE: Socket timeout after 30 seconds. [19:53:17] 2013/08/11 19:46 CRIT z-dat-s7-a MySQL Can't connect to MySQL server on 'z-dat-s7-a' (110) [19:54:16] 2013/08/11 19:53 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [19:54:16] 2013/08/11 19:53 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:54:16] 2013/08/11 19:47 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [19:54:16] 2013/08/11 19:53 OK z-dat-s3-a Load avg. OK - load average: 0.34, 1.02, 2.08 [19:54:16] 2013/08/11 19:53 OK z-dat-s3-a MySQL Uptime: 6246 Threads: 47 Questions: 2484292 Slow queries: 1918 Opens: 30230 Flush tables: 1 Open tables: 10608 Queries per second avg: 397.741 [19:54:16] 2013/08/11 19:53 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63757 MB (10% inode=99%): [19:54:17] 2013/08/11 19:53 OK z-dat-s4-a Load avg. OK - load average: 0.30, 1.06, 2.12 [19:54:17] 2013/08/11 19:53 OK z-dat-s4-a MySQL Uptime: 1416533 Threads: 1 Questions: 54930072 Slow queries: 30 Opens: 129 Flush tables: 1 Open tables: 96 Queries per second avg: 38.777 [19:54:18] 2013/08/11 19:53 OK z-dat-s6-a MySQL Uptime: 32597 Threads: 4 Questions: 2765309 Slow queries: 5764 Opens: 785 Flush tables: 1 Open tables: 567 Queries per second avg: 84.833 [19:54:18] 2013/08/11 19:53 OK z-dat-s7-a MySQL Uptime: 32593 Threads: 28 Questions: 14390914 Slow queries: 1942 Opens: 136955 Flush tables: 1 Open tables: 1169 Queries per second avg: 441.533 [19:55:16] 2013/08/11 19:54 OK hyacinth SMTP SMTP OK - 0.090 sec. response time [19:55:16] 2013/08/11 19:54 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:55:16] 2013/08/11 19:54 OK z-dat-s3-a /tmp DISK OK - free space: /tmp 669 MB (99% inode=99%): [20:11:17] 2013/08/11 20:10 OK nightshade aliasd TCP OK - 0.002 second response time on port 984 [500 Not found.] [20:15:17] 2013/08/11 20:14 OK z-dat-s1-b MySQL slave Uptime: 41204 Threads: 11 Questions: 6339040 Slow queries: 401 Opens: 451 Flush tables: 1 Open tables: 205 Queries per second avg: 153.845 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121 [20:20:17] 2013/08/11 20:18 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [20:21:17] 2013/08/11 20:20 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63736 MB (10% inode=99%): [20:24:17] 2013/08/11 20:22 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3391 [20:25:17] 2013/08/11 20:24 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3528 [20:31:17] 2013/08/11 20:30 OK z-dat-s4-a MySQL slave Uptime: 1418733 Threads: 3 Questions: 55220341 Slow queries: 30 Opens: 129 Flush tables: 1 Open tables: 96 Queries per second avg: 38.922 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1639 [20:33:17] 2013/08/11 20:31 OK z-dat-s6-a MySQL slave Uptime: 34908 Threads: 14 Questions: 3185078 Slow queries: 6000 Opens: 884 Flush tables: 1 Open tables: 570 Queries per second avg: 91.242 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [20:42:18] 2013/08/11 20:41 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:43:18] 2013/08/11 20:42 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63718 MB (10% inode=99%): [20:45:19] 2013/08/11 20:41 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [20:53:19] 2013/08/11 20:52 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3571 [20:58:19] 2013/08/11 20:57 OK z-dat-s3-a MySQL Uptime: 971 Threads: 21 Questions: 532 Slow queries: 4 Opens: 150 Flush tables: 1 Open tables: 84 Queries per second avg: 0.547 [20:58:19] 2013/08/11 20:57 OK z-dat-s3-a MySQL slave Uptime: 971 Threads: 19 Questions: 534 Slow queries: 4 Opens: 150 Flush tables: 1 Open tables: 84 Queries per second avg: 0.549 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [21:02:19] 2013/08/11 20:58 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9129 [21:06:20] 2013/08/11 21:05 OK z-dat-s7-a MySQL slave Uptime: 36939 Threads: 14 Questions: 15094595 Slow queries: 2922 Opens: 137093 Flush tables: 1 Open tables: 1222 Queries per second avg: 408.635 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1731 [21:26:20] 2013/08/11 21:19 CRIT z-dat-s1-b MySQL Can't connect to MySQL server on 'z-dat-s1-b' (111) [21:26:20] 2013/08/11 21:19 CRIT z-dat-s1-b MySQL slave Can't connect to MySQL server on 'z-dat-s1-b' (111) [21:52:21] 2013/08/11 21:50 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [21:54:21] 2013/08/11 21:53 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63669 MB (10% inode=99%): [21:58:21] 2013/08/11 21:56 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3454 [22:04:22] 2013/08/11 22:03 OK z-dat-s3-a MySQL slave Uptime: 4910 Threads: 14 Questions: 5607437 Slow queries: 265 Opens: 23212 Flush tables: 1 Open tables: 14328 Queries per second avg: 1142.44 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1746 [23:53:29] 2013/08/11 23:52 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:57:29] 2013/08/11 23:52 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (110) [23:57:29] 2013/08/11 23:52 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (110) [23:58:29] 2013/08/11 23:57 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 63572 MB (10% inode=99%):