[01:09:04] [[Special:Log/newusers]] create 10 * Selina251 * (New user account) [01:45:31] 2013/09/10 01:44 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3562 [01:46:21] [[Main Page/lang]] ! 10https://wiki.toolserver.org/w/index.php?diff=8140&oldid=8034&rcid=22234 * 5.133.145.82 * (+39) () [01:46:30] 2013/09/10 01:45 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3517.000000 [02:03:30] 2013/09/10 02:02 OK rosemary MySQL slave Uptime: 6022444 Threads: 13 Questions: 4273255227 Slow queries: 625621 Opens: 235203 Flush tables: 1 Open tables: 3747 Queries per second avg: 709.554 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1762 [02:03:30] 2013/09/10 02:02 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1792.000000 [02:59:33] 2013/09/10 02:52 WARN wolfsbane / DISK WARNING - free space: / 6208 MB (20% inode=93%): [02:59:33] 2013/09/10 02:53 WARN wolfsbane /tmp DISK WARNING - free space: / 6208 MB (20% inode=93%): [03:06:33] 2013/09/10 03:06 OK wolfsbane /tmp DISK OK - free space: / 6437 MB (21% inode=93%): [03:07:33] 2013/09/10 03:06 OK wolfsbane / DISK OK - free space: / 6331 MB (21% inode=93%): [04:25:40] 2013/09/10 04:20 CRIT ortelius /tmp DISK CRITICAL - free space: /tmp 973 MB (4% inode=99%): [05:47:42] 2013/09/10 05:46 CRIT ortelius Sun Grid Engine execd Timeout while attempting connection [05:53:42] 2013/09/10 05:46 CRIT ortelius / Timeout while attempting connection [05:53:43] 2013/09/10 05:46 CRIT ortelius DiskSuite Timeout while attempting connection [05:53:43] 2013/09/10 05:46 CRIT ortelius Load avg. Timeout while attempting connection [05:53:43] 2013/09/10 05:46 CRIT ortelius SSH CRITICAL - Socket timeout after 10 seconds [05:53:43] 2013/09/10 05:46 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [05:54:43] 2013/09/10 05:47 CRIT ortelius Environment IPMI Timeout while attempting connection [06:12:42] 2013/09/10 06:05 CRIT ortelius SMTP CRITICAL - Socket timeout after 10 seconds [07:24:45] 2013/09/10 07:17 CRIT ortelius NTP CRITICAL - Socket timeout after 10 seconds [07:39:46] 2013/09/10 07:39 OK ortelius / DISK OK - free space: / 13102 MB (43% inode=91%): [07:39:46] 2013/09/10 07:39 OK ortelius Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [07:39:46] 2013/09/10 07:39 OK ortelius SMTP SMTP OK - 0.007 sec. response time [07:39:46] 2013/09/10 07:39 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [07:40:46] 2013/09/10 07:39 OK ortelius DiskSuite OK - No disk failures detected [07:40:46] 2013/09/10 07:39 OK ortelius Load avg. OK - load average: 2.57, 0.68, 0.25 [07:40:46] 2013/09/10 07:39 OK ortelius NTP NTP OK: Offset -0.00333 secs [07:40:46] 2013/09/10 07:39 OK ortelius SSH SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:40:46] 2013/09/10 07:39 WARN ortelius Sun Grid Engine execd NRPE: Unable to read output [07:41:46] 2013/09/10 07:41 OK ortelius /tmp DISK OK - free space: /tmp 19951 MB (98% inode=99%): [08:11:48] 2013/09/10 08:04 WARN rosemary MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1925 [08:11:48] 2013/09/10 08:05 WARN rosemary s1 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1929.000000 [09:22:51] 2013/09/10 09:22 OK nightshade aliasd TCP OK - 0.003 second response time on port 984 [500 Not found.] [10:19:55] 2013/09/10 10:19 OK rosemary s1 replag QUERY OK: 'SELECT ts_rc_age()' returned 1753.000000 [10:20:56] 2013/09/10 10:19 OK rosemary MySQL slave Uptime: 6052264 Threads: 13 Questions: 4301633055 Slow queries: 628757 Opens: 236342 Flush tables: 1 Open tables: 3789 Queries per second avg: 710.747 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1717 [11:21:58] 2013/09/10 11:20 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3242 [11:23:58] 2013/09/10 11:22 OK z-dat-s6-a MySQL slave Uptime: 523050 Threads: 8 Questions: 153134191 Slow queries: 21584 Opens: 215150 Flush tables: 1 Open tables: 3086 Queries per second avg: 292.771 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1713 [15:52:08] 2013/09/10 15:48 CRIT ortelius Load avg. CRITICAL - load average: 26.13, 26.96, 14.88 [15:53:08] 2013/09/10 15:52 WARN ortelius Load avg. WARNING - load average: 13.16, 23.25, 14.34 [15:56:08] 2013/09/10 15:55 OK ortelius Load avg. OK - load average: 2.59, 13.72, 12.12 [16:37:11] 2013/09/10 16:36 WARN ortelius Load avg. WARNING - load average: 4.24, 21.76, 16.61 [16:40:11] 2013/09/10 16:39 OK ortelius Load avg. OK - load average: 1.84, 12.72, 13.91 [18:46:20] 2013/09/10 18:39 WARN wolfsbane / DISK WARNING - free space: / 4845 MB (16% inode=93%): [18:47:20] 2013/09/10 18:40 WARN wolfsbane /tmp DISK WARNING - free space: / 4680 MB (15% inode=93%): [18:53:21] 2013/09/10 18:52 CRIT wolfsbane / DISK CRITICAL - free space: / 3240 MB (10% inode=93%): [18:54:21] 2013/09/10 18:53 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3069 MB (10% inode=93%): [19:20:19] @replag [19:20:19] Betacommand: s1-rr-a-wd: error; s1-user: 5m 40s [-0.01 s/s]; s2-rr: 14s [-0.00 s/s]; s2-user: 14s [-0.00 s/s]; s2-user-c: error; s3-user: 31s [-0.01 s/s]; s3-user-wd: 8w 4d 59m 32s [+1.00 s/s]; s4-user-wd: 4w 3d 6h 16m 2s [+1.00 s/s] [19:20:20] Betacommand: s6-user-wd: 4w 3d 6h 16m 2s [+1.00 s/s]; s7-user-wd: 4w 3d 6h 16m 2s [+1.00 s/s] [19:26:24] 2013/09/10 19:19 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 7.786 second response time [19:27:24] 2013/09/10 19:26 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.018 second response time [19:35:24] 2013/09/10 19:34 OK wolfsbane / DISK OK - free space: / 13693 MB (45% inode=93%): [19:36:24] 2013/09/10 19:35 OK wolfsbane /tmp DISK OK - free space: / 13688 MB (45% inode=93%): [20:41:05] Marlen Caemmerer * [Toolserver-announce] Maintenance for DB cluster s2 : Resetup of commons [20:45:29] 2013/09/10 20:41 CRIT z-dat-s4-a MySQL slave Slave IO: No Slave SQL: No Seconds Behind Master: (null) [23:23:29] [[Special:Log/newusers]] create 10 * Lois55 * (New user account) [23:35:04] DaB. * Re: [Toolserver-l] Maintenance for DB cluster s2 : Resetup of commons