[03:07:08] 2013/11/13 03:06 OK wolfsbane / DISK OK - free space: / 6325 MB (21% inode=93%): [03:07:09] 2013/11/13 03:06 OK wolfsbane /tmp DISK OK - free space: / 6336 MB (21% inode=93%): [04:30:13] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=8214&oldid=8206&rcid=22434 * 185.15.59.202 * (+0) (updated page) [07:04:21] 2013/11/13 07:03 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3163 [07:04:21] 2013/11/13 07:03 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3060.000000 [07:08:21] 2013/11/13 07:07 OK z-dat-s5-b MySQL slave Uptime: 2312245 Threads: 5 Questions: 2947805702 Slow queries: 18126 Opens: 66907 Flush tables: 1 Open tables: 256 Queries per second avg: 1274.867 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1590 [07:08:21] 2013/11/13 07:07 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1573.000000 [13:07:42] 2013/11/13 13:00 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [13:07:42] 2013/11/13 13:00 CRIT z-dat-s3-a MySQL slave Can't connect to MySQL server on 'z-dat-s3-a' (111) [13:21:43] 2013/11/13 13:20 OK z-dat-s3-a MySQL Uptime: 1235 Threads: 8 Questions: 4408 Slow queries: 1 Opens: 352 Flush tables: 1 Open tables: 340 Queries per second avg: 3.569 [13:21:43] 2013/11/13 13:20 OK z-dat-s3-a MySQL slave Uptime: 1235 Threads: 8 Questions: 4404 Slow queries: 1 Opens: 351 Flush tables: 1 Open tables: 339 Queries per second avg: 3.565 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1383 [13:32:44] 2013/11/13 13:26 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2107 [13:56:47] 2013/11/13 13:56 OK z-dat-s2-b MySQL slave Uptime: 5110044 Threads: 7 Questions: 2047627367 Slow queries: 38504 Opens: 17458665 Flush tables: 1 Open tables: 256 Queries per second avg: 400.706 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1709 [15:40:54] 2013/11/13 15:34 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2096 [16:20:57] 2013/11/13 16:20 CRIT z-dat-s2-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609 [16:44:58] 2013/11/13 16:38 WARN z-dat-s2-b Load avg. WARNING - load average: 14.89, 15.31, 12.79 [16:45:58] 2013/11/13 16:45 OK z-dat-s2-b Load avg. OK - load average: 13.14, 14.70, 12.73 [16:48:17] I'm continuosly getting e-mails from cron via delinker@hawthorn with "/sge/GE/bin/sol-amd64/qcronsub: fork: Not enough space" as of a few days [16:48:36] I suspect it isn't the quota on the account since the home directory is < 100M at the moment. [16:48:50] something on hawthorn itself? [16:58:58] 2013/11/13 16:58 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3346 [17:08:59] 2013/11/13 17:08 OK z-dat-s2-b MySQL slave Uptime: 5121564 Threads: 4 Questions: 2048823207 Slow queries: 38968 Opens: 17500073 Flush tables: 1 Open tables: 256 Queries per second avg: 400.38 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1652 [18:25:01] 2013/11/13 18:18 WARN wolfsbane / DISK WARNING - free space: / 6272 MB (20% inode=93%): [18:25:01] 2013/11/13 18:18 WARN wolfsbane /tmp DISK WARNING - free space: / 6273 MB (20% inode=93%): [18:40:04] Krinkle * [Toolserver-l] Cron <@hawthorn> failures [19:26:43] [[Special:Log/newusers]] create 10 * Jwr12 * (New user account) [21:28:10] 2013/11/13 21:21 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1825 [21:28:11] 2013/11/13 21:21 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1811.000000 [21:29:10] 2013/11/13 21:28 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1799.000000 [21:30:11] 2013/11/13 21:29 OK z-dat-s5-b MySQL slave Uptime: 2363965 Threads: 5 Questions: 3038087317 Slow queries: 18684 Opens: 68292 Flush tables: 1 Open tables: 256 Queries per second avg: 1285.165 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780