[03:10:44] 2013/10/08 03:09 CRIT cassia /sql/data/dewiki DISK CRITICAL - free space: /sql 129998 MB (10% inode=99%): [03:15:44] 2013/10/08 03:09 WARN cassia /sql DISK WARNING - free space: /sql 128734 MB (10% inode=99%): [03:32:47] 2013/10/08 03:26 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2034 [03:32:47] 2013/10/08 03:26 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2052.000000 [03:39:47] 2013/10/08 03:36 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1904 [03:56:47] 2013/10/08 03:50 CRIT z-dat-s5-b MySQL slave Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [04:11:48] 2013/10/08 04:11 CRIT daphne MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [04:11:48] 2013/10/08 04:11 CRIT daphne s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3626.000000 [04:21:49] 2013/10/08 04:20 CRIT z-dat-s4-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3641 [04:26:49] 2013/10/08 04:20 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2216.000000 [04:44:50] 2013/10/08 04:44 OK cassia /sql DISK OK - free space: /sql 143716 MB (12% inode=99%): [04:44:50] 2013/10/08 04:44 WARN cassia /sql/data/dewiki DISK WARNING - free space: /sql 143716 MB (12% inode=99%): [04:50:51] 2013/10/08 04:50 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3656.000000 [05:36:57] [[Special:Log/newusers]] create 10 * Enid156 * (New user account) [07:00:01] 2013/10/08 06:59 CRIT z-dat-s4-a /sql DISK CRITICAL - free space: /sql 36694 MB (5% inode=99%): [08:21:04] 2013/10/08 08:14 CRIT z-dat-s5-b MySQL Too many connections [10:22:10] 2013/10/08 10:21 OK z-dat-s5-b MySQL Uptime: 2500981 Threads: 351 Questions: 4000761162 Slow queries: 14528 Opens: 631421 Flush tables: 1 Open tables: 256 Queries per second avg: 1599.676 [10:29:10] 2013/10/08 10:22 CRIT z-dat-s5-b MySQL Too many connections [11:04:13] 2013/10/08 11:03 OK z-dat-s5-b MySQL slave Uptime: 17 Threads: 1 Questions: 6 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 10 Queries per second avg: 0.352 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [11:05:13] 2013/10/08 11:04 OK z-dat-s5-b MySQL Uptime: 55 Threads: 3 Questions: 1340 Slow queries: 0 Opens: 54 Flush tables: 1 Open tables: 42 Queries per second avg: 24.363 [11:06:13] 2013/10/08 11:05 OK yarrow aliasd TCP OK - 0.001 second response time on port 984 [500 Not found.] [11:06:13] 2013/10/08 11:06 CRIT z-dat-s5-b APT Connection refused by host [11:09:14] 2013/10/08 11:05 CRIT z-dat-s5-b /tmp Connection refused by host [11:11:14] 2013/10/08 11:04 CRIT z-dat-s5-b MySQL slave Can't connect to MySQL server on 'z-dat-s5-b' (111) [11:12:14] 2013/10/08 11:05 CRIT z-dat-s5-b SMTP Connection refused [11:13:14] 2013/10/08 11:06 CRIT z-dat-s5-b / Connection refused by host [11:13:14] 2013/10/08 11:06 CRIT z-dat-s5-b Load avg. Connection refused by host [11:13:14] 2013/10/08 11:06 CRIT z-dat-s5-b MySQL Can't connect to MySQL server on 'z-dat-s5-b' (111) [11:24:14] 2013/10/08 11:23 CRIT z-dat-s5-b PING CRITICAL - Host Unreachable (z-dat-s5-b) [11:24:14] 2013/10/08 11:23 CRIT z-dat-s5-b SSH No route to host [11:28:14] 2013/10/08 11:27 CRIT sage APT Connection refused by host [11:29:14] 2013/10/08 11:28 CRIT sage PING CRITICAL - Host Unreachable (sage) [11:30:14] 2013/10/08 11:29 CRIT sage / Connection refused or timed out [11:30:15] 2013/10/08 11:29 CRIT sage /tmp Connection refused or timed out [11:30:15] 2013/10/08 11:29 CRIT sage Environment IPMI Connection refused or timed out [11:30:15] 2013/10/08 11:29 CRIT sage Load avg. Connection refused or timed out [11:30:15] 2013/10/08 11:29 CRIT sage NTP CRITICAL - Socket timeout after 10 seconds [11:30:15] 2013/10/08 11:29 CRIT sage SMTP No route to host [11:30:15] 2013/10/08 11:29 CRIT sage SSH No route to host [11:32:15] 2013/10/08 11:31 OK sage PING PING OK - Packet loss = 16%, RTA = 0.17 ms [11:33:15] 2013/10/08 11:32 OK sage / DISK OK - free space: / 1503 MB (84% inode=92%): [11:33:15] 2013/10/08 11:32 OK sage /tmp DISK OK - free space: /tmp 4317 MB (96% inode=99%): [11:33:15] 2013/10/08 11:32 WARN sage APT APT WARNING: 2 packages available for upgrade (0 critical updates). [11:33:15] 2013/10/08 11:32 OK sage Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [11:33:15] 2013/10/08 11:32 OK sage Load avg. OK - load average: 0.19, 0.08, 0.03 [11:33:16] 2013/10/08 11:32 OK sage SMTP SMTP OK - 0.170 sec. response time [11:33:16] 2013/10/08 11:32 OK sage SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [11:34:15] 2013/10/08 11:33 CRIT z-dat-s5-b CRITICAL - Host Unreachable (z-dat-s5-b) [11:39:15] 2013/10/08 11:33 WARN sage NTP NTP WARNING: Server has the LI_ALARM bit set, Offset 5.5e-05 secs [11:50:16] 2013/10/08 11:49 OK sage NTP NTP OK: Offset -0.002253 secs [12:37:21] 2013/10/08 12:36 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3269 [12:41:21] 2013/10/08 12:40 OK z-dat-s4-a MySQL slave Uptime: 2946914 Threads: 1 Questions: 203071984 Slow queries: 109 Opens: 328 Flush tables: 1 Open tables: 249 Queries per second avg: 68.910 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1451 [18:49:05] DaB. * Re: [Toolserver-l] Need Tool Labs migration support? [18:56:47] [[Special:Log/newusers]] create 10 * Armando Pacheco * (New user account) [19:59:44] 2013/10/08 19:59 OK z-dat-s5-b PING OK - Packet loss = 0%, RTA = 0.15 ms [19:59:44] 2013/10/08 19:59 OK z-dat-s5-b / DISK OK - free space: / 2587 MB (67% inode=88%): [19:59:44] 2013/10/08 19:59 OK z-dat-s5-b Load avg. OK - load average: 0.54, 0.11, 0.04 [19:59:44] 2013/10/08 19:59 OK z-dat-s5-b PING PING OK - Packet loss = 0%, RTA = 0.14 ms [19:59:44] 2013/10/08 19:59 OK z-dat-s5-b SMTP SMTP OK - 0.111 sec. response time [20:00:44] 2013/10/08 20:00 OK z-dat-s5-b /sql DISK OK - free space: / 2587 MB (67% inode=88%): [20:00:44] 2013/10/08 19:59 OK z-dat-s5-b /tmp DISK OK - free space: / 2587 MB (67% inode=88%): [20:00:44] 2013/10/08 20:00 WARN z-dat-s5-b APT APT WARNING: 2 packages available for upgrade (0 critical updates). [20:00:44] 2013/10/08 19:59 OK z-dat-s5-b SSH SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [20:05:44] 2013/10/08 20:05 OK z-dat-s5-b MySQL Uptime: 40 Threads: 1 Questions: 2166 Slow queries: 0 Opens: 40 Flush tables: 1 Open tables: 33 Queries per second avg: 54.150 [20:07:44] 2013/10/08 20:01 CRIT z-dat-s5-b /sql DISK CRITICAL - free space: /sql 8475 MB (0% inode=99%): [21:53:52] 2013/10/08 21:49 CRIT z-dat-s7-a MySQL slave Can't connect to MySQL server on 'z-dat-s7-a' (110) [21:54:53] 2013/10/08 21:53 OK z-dat-s7-a MySQL slave Uptime: 1845545 Threads: 7 Questions: 394703022 Slow queries: 21799 Opens: 1815729 Flush tables: 1 Open tables: 6046 Queries per second avg: 213.868 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4 [22:21:05] Kai Krueger * Re: [Toolserver-l] [Maps-l] Making Toolserver work - rate limited OSM [22:25:54] 2013/10/08 22:21 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [22:26:54] 2013/10/08 22:25 OK z-dat-s6-a MySQL Uptime: 2982044 Threads: 6 Questions: 909338018 Slow queries: 79505 Opens: 946883 Flush tables: 1 Open tables: 4134 Queries per second avg: 304.937 [22:42:54] 2013/10/08 22:42 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3414 [22:42:54] 2013/10/08 22:42 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3324.000000 [22:46:05] Magnus Manske * Re: [Toolserver-l] Need Tool Labs migration support? [22:47:54] 2013/10/08 22:47 OK daphne MySQL slave Uptime: 14958034 Threads: 7 Questions: 2299963838 Slow queries: 1520716 Opens: 485595 Flush tables: 1 Open tables: 2008 Queries per second avg: 153.761 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1560 [22:47:54] 2013/10/08 22:47 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1366.000000 [23:10:54] 2013/10/08 23:10 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3472 [23:11:54] 2013/10/08 23:10 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3339.000000 [23:15:55] 2013/10/08 23:15 OK z-dat-s5-b MySQL slave Uptime: 11452 Threads: 3 Questions: 87971954 Slow queries: 313 Opens: 521 Flush tables: 1 Open tables: 255 Queries per second avg: 7681.798 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1548 [23:15:55] 2013/10/08 23:14 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1738.000000 [23:17:05] DaB. * Re: [Toolserver-l] Need Tool Labs migration support?