[00:43:37] 2013/07/08 00:36 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1912 [00:55:38] 2013/07/08 00:49 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1904.000000 [01:38:40] 2013/07/08 01:37 OK z-dat-s2-b MySQL slave Uptime: 4295675 Threads: 9 Questions: 3935855475 Slow queries: 78234 Opens: 31175459 Flush tables: 1 Open tables: 256 Queries per second avg: 916.236 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1797 [01:43:41] 2013/07/08 01:42 OK wolfsbane Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [01:43:41] 2013/07/08 01:43 OK wolfsbane Load avg. OK - load average: 3.39, 1.01, 0.37 [01:43:41] 2013/07/08 01:42 OK wolfsbane NTP NTP OK: Offset -0.0001 secs [01:43:41] 2013/07/08 01:42 OK wolfsbane SMTP SMTP OK - 9.003 sec. response time [01:43:41] 2013/07/08 01:42 OK wolfsbane SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:43:42] 2013/07/08 01:42 WARN wolfsbane Sun Grid Engine execd NRPE: Unable to read output [01:43:42] 2013/07/08 01:42 OK wolfsbane toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [01:50:41] 2013/07/08 01:50 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1744.000000 [02:07:04] Avocato * [Toolserver-l] Three problems [03:07:44] 2013/07/08 03:07 WARN wolfsbane /tmp DISK WARNING - free space: / 3706 MB (12% inode=92%): [03:08:44] 2013/07/08 03:07 WARN wolfsbane / DISK WARNING - free space: / 3701 MB (12% inode=92%): [04:08:48] 2013/07/08 04:07 CRIT wolfsbane / DISK CRITICAL - free space: / 3282 MB (10% inode=92%): [04:08:48] 2013/07/08 04:08 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3243 MB (10% inode=92%): [04:13:48] 2013/07/08 04:12 OK wolfsbane / DISK OK - free space: / 10595 MB (35% inode=92%): [04:13:48] 2013/07/08 04:13 OK wolfsbane /tmp DISK OK - free space: / 10595 MB (35% inode=92%): [04:51:51] 2013/07/08 04:50 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [04:52:51] 2013/07/08 04:51 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 66208 MB (10% inode=99%): [05:48:00] [[Special:Log/newusers]] create 10 * Dandv * (New user account) [05:48:48] [[Category:Edit counters]] ! 10https://wiki.toolserver.org/w/index.php?diff=8065&oldid=8006&rcid=22033 * Dandv * (-1) (rm extra (, move the useful info to the top) [05:50:29] [[~escaladix/larticles/larticles.php]] ! 10https://wiki.toolserver.org/w/index.php?diff=8066&oldid=6967&rcid=22034 * Dandv * (+180) (Point to a working tool) [06:29:58] 2013/07/08 06:27 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2131.000000 [07:53:04] Marlen Caemmerer * Re: [Toolserver-l] Three problems [08:37:06] 2013/07/08 08:36 CRIT z-dat-s2-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3635.000000 [08:51:07] 2013/07/08 08:50 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3559.000000 [09:05:07] 2013/07/08 09:04 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:10:07] 2013/07/08 09:04 ?? hyacinth / CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:10:07] 2013/07/08 09:04 CRIT z-dat-s3-a / Connection refused by host [09:10:07] 2013/07/08 09:04 ?? z-dat-s7-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:07] 2013/07/08 09:11 OK hyacinth / DISK OK - free space: / 6336 MB (21% inode=84%): [09:11:07] 2013/07/08 09:04 ?? hyacinth /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:07] 2013/07/08 09:04 ?? hyacinth RAID CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:07] 2013/07/08 09:04 CRIT hyacinth SSH CRITICAL - Socket timeout after 10 seconds [09:11:07] 2013/07/08 09:04 CRIT z-dat-s3-a /sql Connection refused by host [09:11:08] 2013/07/08 09:04 CRIT z-dat-s3-a /tmp Connection refused by host [09:11:08] 2013/07/08 09:04 CRIT z-dat-s3-a Load avg. Connection refused by host [09:11:09] 2013/07/08 09:10 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 66105 MB (10% inode=99%): [09:11:09] 2013/07/08 09:04 ?? z-dat-s4-a Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:10] 2013/07/08 09:04 ?? z-dat-s6-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:10] 2013/07/08 09:04 ?? z-dat-s6-a /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:11] 2013/07/08 09:10 OK z-dat-s7-a /sql DISK OK - free space: /sql 138495 MB (19% inode=99%): [09:11:11] 2013/07/08 09:04 ?? z-dat-s7-a /tmp CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:12] 2013/07/08 09:04 ?? z-dat-s7-a Load avg. CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:12:07] 2013/07/08 09:11 OK hyacinth /tmp DISK OK - free space: /tmp 40855 MB (99% inode=99%): [09:12:07] 2013/07/08 09:11 OK hyacinth RAID OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:12:07] 2013/07/08 09:11 OK hyacinth SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:12:07] 2013/07/08 09:11 OK z-dat-s4-a Load avg. OK - load average: 3.14, 2.58, 2.62 [09:12:07] 2013/07/08 09:11 OK z-dat-s6-a /sql DISK OK - free space: /sql 301773 MB (37% inode=99%): [09:12:08] 2013/07/08 09:11 OK z-dat-s6-a /tmp DISK OK - free space: /tmp 40891 MB (99% inode=99%): [09:12:08] 2013/07/08 09:11 OK z-dat-s7-a /tmp DISK OK - free space: /tmp 40854 MB (99% inode=99%): [09:12:09] 2013/07/08 09:11 OK z-dat-s7-a Load avg. OK - load average: 3.14, 2.56, 2.61 [09:14:07] 2013/07/08 09:10 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (111) [09:17:07] 2013/07/08 09:10 CRIT z-dat-s6-a MySQL slave Can't connect to MySQL server on 'z-dat-s6-a' (111) [09:17:07] 2013/07/08 09:10 CRIT z-dat-s6-a wikidata replag QUERY CRITICAL: Can't connect to MySQL server on 'z-dat-s6-a' (111) [09:22:08] 2013/07/08 09:21 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1694.000000 [09:24:08] 2013/07/08 09:23 OK z-dat-s6-a MySQL Uptime: 776 Threads: 1 Questions: 2 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 8 Queries per second avg: 0.2 [09:24:08] 2013/07/08 09:23 OK z-dat-s6-a MySQL slave Uptime: 785 Threads: 2 Questions: 186 Slow queries: 0 Opens: 55 Flush tables: 1 Open tables: 46 Queries per second avg: 0.236 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 791 [09:24:08] 2013/07/08 09:23 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 794.000000 [09:47:10] 2013/07/08 09:40 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2172.000000 [09:51:10] 2013/07/08 09:50 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1737.000000 [10:10:12] 2013/07/08 10:03 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2077.000000 [10:28:14] 2013/07/08 10:27 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1763.000000 [12:16:04] Sumana Harihareswara * Re: [Toolserver-l] Three problems [12:26:18] 2013/07/08 12:19 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.839 second response time [12:27:18] 2013/07/08 12:26 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.342 second response time [12:40:20] 2013/07/08 12:37 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1983 [12:43:20] 2013/07/08 12:37 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2065.000000 [12:45:20] 2013/07/08 12:39 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1947.000000 [13:12:21] 2013/07/08 13:12 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 [13:12:21] 2013/07/08 13:12 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3602.000000 [13:18:21] 2013/07/08 13:17 CRIT z-dat-s5-b s4 replag (Service Check Timed Out) [13:30:22] 2013/07/08 13:23 WARN ptolemy Load avg. WARNING - load average: 19.00, 16.44, 10.56 [13:43:22] 2013/07/08 13:42 OK ptolemy Load avg. OK - load average: 10.12, 13.94, 12.87 [14:43:38] [[Special:Log/newusers]] create 10 * LeelaBratee * (New user account) [14:45:29] [[Special:Log/newusers]] create2 10 * LeelaBratee * (created new account User:Leela Bratee) [15:17:29] 2013/07/08 15:16 OK z-dat-s4-a /sql DISK OK - free space: /sql 67105 MB (11% inode=99%): [15:24:29] 2013/07/08 15:17 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 66716 MB (10% inode=99%): [15:32:29] 2013/07/08 15:32 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3400.000000 [15:39:29] 2013/07/08 15:39 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3506 [15:39:29] 2013/07/08 15:39 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3506.000000 [15:41:29] 2013/07/08 15:41 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1637.000000 [15:46:30] 2013/07/08 15:46 OK z-dat-s5-b MySQL slave Uptime: 1305391 Threads: 6 Questions: 1713113042 Slow queries: 7680 Opens: 734589 Flush tables: 1 Open tables: 255 Queries per second avg: 1312.337 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1722 [15:46:30] 2013/07/08 15:46 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1723.000000 [15:53:30] 2013/07/08 15:47 CRIT wolfsbane Load avg. CRITICAL - load average: 12.53, 27.55, 17.57 [15:54:30] 2013/07/08 15:53 WARN wolfsbane Load avg. WARNING - load average: 6.65, 23.18, 16.66 [15:57:30] 2013/07/08 15:56 OK wolfsbane Load avg. OK - load average: 2.57, 14.02, 14.17 [17:16:37] 2013/07/08 17:10 WARN wolfsbane /tmp DISK WARNING - free space: / 6247 MB (20% inode=92%): [17:18:37] 2013/07/08 17:11 WARN wolfsbane / DISK WARNING - free space: / 6238 MB (20% inode=92%): [17:49:40] 2013/07/08 17:43 CRIT ortelius Load avg. CRITICAL - load average: 17.25, 29.91, 17.46 [17:49:40] 2013/07/08 17:49 OK wolfsbane /tmp DISK OK - free space: / 9965 MB (33% inode=92%): [17:50:40] 2013/07/08 17:49 OK wolfsbane / DISK OK - free space: / 9971 MB (33% inode=92%): [17:52:40] 2013/07/08 17:51 WARN ortelius Load avg. WARNING - load average: 6.24, 21.48, 16.55 [17:55:40] 2013/07/08 17:54 OK ortelius Load avg. OK - load average: 2.43, 12.96, 14.03 [18:03:42] 2013/07/08 18:00 WARN ortelius Load avg. WARNING - load average: 4.03, 17.06, 17.13 [18:04:41] 2013/07/08 18:03 WARN wolfsbane Load avg. WARNING - load average: 7.12, 19.91, 16.55 [18:06:41] 2013/07/08 18:05 OK ortelius Load avg. OK - load average: 1.04, 9.85, 14.23 [18:06:41] 2013/07/08 18:06 OK wolfsbane Load avg. OK - load average: 2.11, 13.83, 14.67 [20:11:01] [[Special:Log/newusers]] create 10 * Melyanow * (New user account) [22:32:55] 2013/07/08 22:25 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2124.000000 [22:45:56] 2013/07/08 22:45 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1794.000000