[01:24:02] 2013/09/06 01:17 CRIT web.amaranth journal.toolserver.org HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 154 bytes in 0.584 second response time [01:25:02] 2013/09/06 01:24 OK web.amaranth journal.toolserver.org HTTP OK: HTTP/1.1 200 OK - 213 bytes in 0.772 second response time [01:47:04] 2013/09/06 01:39 WARN wolfsbane / DISK WARNING - free space: / 6288 MB (20% inode=93%): [01:47:04] 2013/09/06 01:40 WARN wolfsbane /tmp DISK WARNING - free space: / 6287 MB (20% inode=93%): [02:02:04] 2013/09/06 01:55 CRIT web.amaranth journal.toolserver.org HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 154 bytes in 0.584 second response time [02:04:04] 2013/09/06 02:03 OK web.amaranth journal.toolserver.org HTTP OK: HTTP/1.1 200 OK - 213 bytes in 0.793 second response time [03:07:07] 2013/09/06 03:06 OK wolfsbane / DISK OK - free space: / 6532 MB (21% inode=93%): [03:07:07] 2013/09/06 03:06 OK wolfsbane /tmp DISK OK - free space: / 6611 MB (22% inode=93%): [03:14:07] 2013/09/06 03:07 CRIT web.amaranth journal.toolserver.org HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 154 bytes in 0.587 second response time [04:53:16] 2013/09/06 04:46 WARN ortelius /tmp DISK WARNING - free space: /tmp 3291 MB (15% inode=99%): [05:08:19] 2013/09/06 05:07 OK ortelius /tmp DISK OK - free space: /tmp 5899 MB (28% inode=99%): [05:20:19] 2013/09/06 05:13 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2193.000000 [05:32:20] 2013/09/06 05:31 OK willow / DISK OK - free space: / 27410 MB (26% inode=99%): [05:32:21] 2013/09/06 05:31 OK willow /tmp DISK OK - free space: / 27410 MB (26% inode=99%): [05:45:21] 2013/09/06 05:44 CRIT rosemary s1 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3616.000000 [05:46:21] 2013/09/06 05:45 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3580.000000 [05:50:21] 2013/09/06 05:45 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3151 [05:56:22] 2013/09/06 05:49 WARN willow / DISK WARNING - free space: / 19026 MB (18% inode=98%): [05:56:22] 2013/09/06 05:49 WARN willow /tmp DISK WARNING - free space: / 19206 MB (18% inode=98%): [06:06:23] 2013/09/06 06:05 OK rosemary MySQL slave Uptime: 5691424 Threads: 8 Questions: 3967419950 Slow queries: 587951 Opens: 222126 Flush tables: 1 Open tables: 3775 Queries per second avg: 697.87 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [06:07:23] 2013/09/06 06:06 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1724.000000 [06:15:24] 2013/09/06 06:14 CRIT willow / DISK CRITICAL - free space: / 11476 MB (10% inode=98%): [06:15:24] 2013/09/06 06:14 CRIT willow /tmp DISK CRITICAL - free space: / 11476 MB (10% inode=98%): [06:21:48] [[Special:Log/newusers]] create 10 * Bennett869 * (New user account) [08:03:10] 2013/09/06 08:01 WARN willow / DISK WARNING - free space: / 12969 MB (12% inode=98%): [08:03:10] 2013/09/06 08:01 WARN willow /tmp DISK WARNING - free space: / 12667 MB (12% inode=98%): [08:07:30] 2013/09/06 08:06 OK web.amaranth journal.toolserver.org HTTP OK: HTTP/1.1 200 OK - 144 bytes in 0.594 second response time [08:09:30] 2013/09/06 08:08 OK willow / DISK OK - free space: / 22673 MB (21% inode=99%): [08:10:30] 2013/09/06 08:09 OK willow /tmp DISK OK - free space: / 22600 MB (21% inode=99%): [08:17:31] 2013/09/06 08:10 WARN web.amaranth journal.toolserver.org HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 481 bytes in 0.486 second response time [09:50:37] 2013/09/06 09:46 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1939 [10:12:38] [[Special:Log/newusers]] create 10 * Kelly165 * (New user account) [10:22:38] 2013/09/06 10:21 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [10:47:45] @replag [10:47:45] Betacommand: s1-rr-a-wd: error; s2-rr: 15s [-0.00 s/s]; s2-user: 15s [-0.00 s/s]; s2-user-c: error; s3-user: 1h 13m 50s [+0.01 s/s]; s3-user-wd: 7w 6d 16h 26m 58s [+1.00 s/s]; s4-user-wd: 3w 5d 21h 43m 28s [+1.00 s/s]; s6-user-wd: 3w 5d 21h 43m 28s [+1.00 s/s] [10:47:46] Betacommand: s7-user-wd: 3w 5d 21h 43m 28s [+1.00 s/s] [11:06:40] 2013/09/06 11:05 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [11:07:40] 2013/09/06 11:06 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 62310 MB (10% inode=99%): [11:15:41] 2013/09/06 11:14 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3520 [11:23:40] 2013/09/06 11:22 OK z-dat-s3-a MySQL slave Uptime: 177447 Threads: 21 Questions: 203812511 Slow queries: 17172 Opens: 905448 Flush tables: 1 Open tables: 16384 Queries per second avg: 1148.582 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1680 [12:14:44] 2013/09/06 12:14 OK yarrow aliasd TCP OK - 0.002 second response time on port 984 [500 Not found.] [15:00:51] 2013/09/06 14:54 WARN wolfsbane /tmp DISK WARNING - free space: / 6277 MB (20% inode=93%): [15:01:51] 2013/09/06 14:54 WARN wolfsbane / DISK WARNING - free space: / 6274 MB (20% inode=93%): [17:06:51] * ToAruShiroiNeko pets legoktm [17:49:50] Hi [17:49:55] Why does dabsolver break? [17:50:08] Says something about a user exceeding connections [17:50:30] Qcoder00: too many database connections at the same time [17:50:41] Right [17:50:47] which means it's popular [17:50:49] :) [17:51:23] But annoying when you know what you are doing and fixing diambigs at speed [17:51:45] Qcoder00: it prevents broken tools from killing the servers [17:51:57] True [17:52:07] How long does it take to clear? [18:00:07] Qcoder00: when a connection closes a spot for a new one opens [18:00:36] OK [18:23:59] 2013/09/06 18:22 CRIT wolfsbane / DISK CRITICAL - free space: / 3262 MB (10% inode=93%): [18:23:59] 2013/09/06 18:23 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3202 MB (10% inode=93%): [20:08:04] 2013/09/06 20:07 WARN wolfsbane /tmp DISK WARNING - free space: / 5785 MB (19% inode=93%): [20:09:04] 2013/09/06 20:07 WARN wolfsbane / DISK WARNING - free space: / 5784 MB (19% inode=93%): [20:24:05] John * [Toolserver-l] Cron [21:42:04] DaB. * Re: [Toolserver-l] Cron [22:49:04] John * Re: [Toolserver-l] Cron