[03:59:22] 2013/07/13 03:58 WARN daphne wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3490.000000 [04:06:21] 2013/07/13 04:05 OK daphne wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1630.000000 [04:11:21] 2013/07/13 04:10 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3457.000000 [04:16:23] 2013/07/13 04:15 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1248.000000 [07:51:51] toolserver webservers down? [09:28:32] looks like it [09:58:04] Platonides * [Toolserver-l] toolserver web down [10:09:53] @replag [10:09:53] Steinsplitter: s2-rr: 1m 40s [+0.00 s/s]; s2-user: 1m 40s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 3m 24s [-0.13 s/s]; s3-user: 1m 53s [+0.00 s/s]; s7-user: 11s [+0.00 s/s] [13:12:52] 2013/07/13 13:06 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2127.000000 [13:16:06] toolserver.org down ? [13:23:14] looks like it [13:23:22] at least webserver [14:03:53] 2013/07/13 14:03 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1790.000000 [14:42:04] DaB. * Re: [Toolserver-l] toolserver web down [15:59:00] 2013/07/13 15:54 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2000 [16:00:00] 2013/07/13 15:52 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2088 [16:02:00] 2013/07/13 15:55 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2179 [16:05:01] 2013/07/13 15:58 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2070.000000 [16:20:02] 2013/07/13 16:12 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2143.000000 [16:28:03] 2013/07/13 16:26 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3637 [16:29:03] 2013/07/13 16:28 CRIT z-dat-s6-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3654 [16:36:05] 2013/07/13 16:34 CRIT z-dat-s7-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [16:48:05] 2013/07/13 16:47 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3607.000000 [17:00:06] 2013/07/13 16:58 CRIT z-dat-s6-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3623.000000 [18:10:11] 2013/07/13 18:09 ?? z-dat-s4-a /sql CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:14:12] 2013/07/13 18:09 CRIT z-dat-s3-a MySQL Can't connect to MySQL server on 'z-dat-s3-a' (111) [18:15:12] 2013/07/13 18:09 ?? hyacinth Environment IPMI CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:15:12] 2013/07/13 18:10 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [18:15:12] 2013/07/13 18:10 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [18:16:12] 2013/07/13 18:15 OK hyacinth Environment IPMI ok: temperature ok fan ok voltage ok chassis ok [18:16:12] 2013/07/13 18:15 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 65951 MB (10% inode=99%): [18:16:12] 2013/07/13 18:15 OK z-dat-s4-a MySQL Uptime: 4009157 Threads: 4 Questions: 227078812 Slow queries: 74 Opens: 387 Flush tables: 1 Open tables: 257 Queries per second avg: 56.640 [18:16:12] 2013/07/13 18:15 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1835 [18:17:12] 2013/07/13 18:16 OK z-dat-s4-a MySQL slave Uptime: 4009205 Threads: 1 Questions: 227085090 Slow queries: 74 Opens: 387 Flush tables: 1 Open tables: 257 Queries per second avg: 56.640 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1792 [18:17:12] 2013/07/13 18:15 OK z-dat-s6-a MySQL Uptime: 119830 Threads: 13 Questions: 78133340 Slow queries: 9040 Opens: 12334 Flush tables: 1 Open tables: 2956 Queries per second avg: 652.34 [18:36:14] 2013/07/13 18:34 OK z-dat-s3-a MySQL Uptime: 1194 Threads: 22 Questions: 37785 Slow queries: 6 Opens: 451 Flush tables: 1 Open tables: 414 Queries per second avg: 31.645 [18:47:14] 2013/07/13 18:46 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3082 [18:50:14] 2013/07/13 18:49 OK z-dat-s7-a MySQL slave Uptime: 206520 Threads: 8 Questions: 155704978 Slow queries: 9664 Opens: 450408 Flush tables: 1 Open tables: 3510 Queries per second avg: 753.946 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1368 [18:52:13] 2013/07/13 18:51 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3362 [18:56:13] 2013/07/13 18:55 OK z-dat-s6-a MySQL slave Uptime: 122189 Threads: 5 Questions: 80818733 Slow queries: 9202 Opens: 12338 Flush tables: 1 Open tables: 2960 Queries per second avg: 661.423 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1595 [19:06:13] 2013/07/13 19:05 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3574.000000 [19:06:14] 2013/07/13 19:05 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3556.000000 [19:23:13] 2013/07/13 19:22 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3480 [19:23:14] 2013/07/13 19:22 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1747.000000 [19:23:14] 2013/07/13 19:22 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1723.000000 [19:28:13] 2013/07/13 19:27 OK z-dat-s3-a MySQL slave Uptime: 4314 Threads: 20 Questions: 6494460 Slow queries: 116 Opens: 27399 Flush tables: 1 Open tables: 16384 Queries per second avg: 1505.438 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1643 [21:33:40] s4 seems corrupted [22:35:04] Platonides * [Toolserver-l] commonswiki_p corrupted [22:42:05] ^ amette