[01:37:06] 2013/07/04 01:30 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2061.000000 [01:38:06] 2013/07/04 01:33 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1983 [01:38:06] 2013/07/04 01:31 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2097 [01:46:06] 2013/07/04 01:39 WARN wolfsbane /tmp DISK WARNING - free space: / 6262 MB (20% inode=92%): [01:47:06] 2013/07/04 01:40 WARN wolfsbane / DISK WARNING - free space: / 6257 MB (20% inode=92%): [01:55:07] 2013/07/04 01:48 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2187 [01:56:07] 2013/07/04 01:49 WARN z-dat-s6-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1982.000000 [02:04:06] 2013/07/04 02:03 CRIT z-dat-s7-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3625 [02:04:06] 2013/07/04 02:03 CRIT z-dat-s7-a wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3653.000000 [02:11:06] 2013/07/04 02:09 CRIT z-dat-s3-a MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3614 [02:42:08] 2013/07/04 02:40 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3447 [02:44:08] 2013/07/04 02:43 OK z-dat-s6-a MySQL slave Uptime: 1049599 Threads: 4 Questions: 700143467 Slow queries: 39537 Opens: 287216 Flush tables: 1 Open tables: 3380 Queries per second avg: 667.58 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1662 [02:44:08] 2013/07/04 02:43 OK z-dat-s6-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1701.000000 [02:55:09] 2013/07/04 02:53 OK z-dat-s3-a MySQL slave Uptime: 50517 Threads: 16 Questions: 66144771 Slow queries: 6471 Opens: 200195 Flush tables: 1 Open tables: 16384 Queries per second avg: 1309.356 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1500 [02:58:09] 2013/07/04 02:57 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3570 [03:01:09] 2013/07/04 03:00 WARN z-dat-s7-a wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3476.000000 [03:04:09] 2013/07/04 03:03 OK z-dat-s7-a MySQL slave Uptime: 51067 Threads: 6 Questions: 28995652 Slow queries: 3949 Opens: 131874 Flush tables: 1 Open tables: 1467 Queries per second avg: 567.796 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1731 [03:07:09] 2013/07/04 03:06 OK wolfsbane /tmp DISK OK - free space: / 6341 MB (21% inode=92%): [03:08:09] 2013/07/04 03:07 OK wolfsbane / DISK OK - free space: / 7557 MB (25% inode=92%): [03:12:10] 2013/07/04 03:11 OK z-dat-s7-a wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1666.000000 [03:46:12] 2013/07/04 03:44 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [03:47:12] 2013/07/04 03:46 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 66780 MB (10% inode=99%): [06:04:20] 2013/07/04 05:59 CRIT z-dat-s4-a MySQL (Service Check Timed Out) [06:05:20] 2013/07/04 05:59 CRIT z-dat-s4-a MySQL slave (Service Check Timed Out) [06:05:20] 2013/07/04 06:00 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [06:06:20] 2013/07/04 06:04 OK z-dat-s4-a MySQL Uptime: 3187776 Threads: 1 Questions: 184947691 Slow queries: 74 Opens: 267 Flush tables: 1 Open tables: 259 Queries per second avg: 58.17 [06:06:20] 2013/07/04 06:04 OK z-dat-s4-a MySQL slave Uptime: 3187776 Threads: 1 Questions: 184947689 Slow queries: 74 Opens: 267 Flush tables: 1 Open tables: 259 Queries per second avg: 58.17 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [06:06:20] 2013/07/04 06:05 OK z-dat-s6-a MySQL Uptime: 1061723 Threads: 7 Questions: 705543085 Slow queries: 40363 Opens: 287293 Flush tables: 1 Open tables: 3405 Queries per second avg: 664.526 [07:40:27] 2013/07/04 07:33 CRIT yarrow aliasd Connection refused [07:51:27] 2013/07/04 07:50 CRIT z-dat-s4-a /sql CHECK_NRPE: Socket timeout after 30 seconds. [07:54:27] 2013/07/04 07:49 CRIT z-dat-s3-a MySQL (Service Check Timed Out) [07:54:27] 2013/07/04 07:49 CRIT z-dat-s3-a MySQL slave (Service Check Timed Out) [07:54:28] 2013/07/04 07:53 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 66531 MB (10% inode=99%): [07:54:28] 2013/07/04 07:49 CRIT z-dat-s6-a MySQL Can't connect to MySQL server on 'z-dat-s6-a' (110) [07:55:27] 2013/07/04 07:54 OK z-dat-s3-a MySQL Uptime: 68513 Threads: 31 Questions: 87167311 Slow queries: 8867 Opens: 394193 Flush tables: 1 Open tables: 16385 Queries per second avg: 1272.274 [07:55:28] 2013/07/04 07:54 OK z-dat-s3-a MySQL slave Uptime: 68513 Threads: 34 Questions: 87167306 Slow queries: 8867 Opens: 394193 Flush tables: 1 Open tables: 16385 Queries per second avg: 1272.273 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 286 [07:55:28] 2013/07/04 07:53 OK z-dat-s6-a MySQL Uptime: 1068283 Threads: 13 Questions: 710080820 Slow queries: 41398 Opens: 287615 Flush tables: 1 Open tables: 3415 Queries per second avg: 664.693 [09:07:32] 2013/07/04 09:00 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 7.072 second response time [09:19:33] 2013/07/04 09:18 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.762 second response time [09:20:33] 2013/07/04 09:19 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [09:24:34] 2013/07/04 09:23 WARN ortelius toolserver.org HTTP HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.839 second response time [09:25:33] 2013/07/04 09:24 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 6.017 second response time [09:27:33] 2013/07/04 09:26 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.287 second response time [10:11:34] 2013/07/04 10:08 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1816 [10:12:34] 2013/07/04 10:12 OK z-dat-s5-b MySQL slave Uptime: 939751 Threads: 8 Questions: 1231326406 Slow queries: 4647 Opens: 450508 Flush tables: 1 Open tables: 256 Queries per second avg: 1310.268 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1746 [11:22:40] 2013/07/04 11:18 CRIT ortelius toolserver.org HTTP CRITICAL - Socket timeout after 10 seconds [11:24:40] 2013/07/04 11:23 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.098 second response time [12:09:41] 2013/07/04 12:02 CRIT ortelius toolserver.org HTTP HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.235 second response time [12:27:42] 2013/07/04 12:26 OK ortelius toolserver.org HTTP HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.365 second response time [12:32:42] 2013/07/04 12:29 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1973 [12:35:42] 2013/07/04 12:28 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2142.000000 [12:36:42] 2013/07/04 12:29 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2199.000000 [13:07:44] 2013/07/04 13:06 CRIT z-dat-s5-b s4 replag (Service Check Timed Out) [13:08:44] 2013/07/04 13:07 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2944.000000 [13:19:44] 2013/07/04 13:18 CRIT z-dat-s5-b s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3601.000000 [13:27:45] 2013/07/04 13:27 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3650 [13:27:45] 2013/07/04 13:26 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3639.000000 [13:31:45] 2013/07/04 13:31 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3518 [13:31:45] 2013/07/04 13:30 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3576.000000 [13:36:45] 2013/07/04 13:36 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3576.000000 [13:40:45] 2013/07/04 13:39 CRIT z-dat-s5-b s4 replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3607.000000 [13:41:46] 2013/07/04 13:40 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3269.000000 [13:43:47] 2013/07/04 13:43 OK z-dat-s5-b MySQL slave Uptime: 952411 Threads: 6 Questions: 1235477288 Slow queries: 4963 Opens: 453945 Flush tables: 1 Open tables: 256 Queries per second avg: 1297.210 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1735 [13:43:47] 2013/07/04 13:42 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1786.000000 [13:51:47] 2013/07/04 13:51 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1728.000000 [14:10:48] 2013/07/04 14:04 WARN ptolemy Load avg. WARNING - load average: 12.67, 15.26, 11.47 [14:11:48] 2013/07/04 14:11 OK ptolemy Load avg. OK - load average: 9.62, 13.95, 11.24 [14:27:49] 2013/07/04 14:20 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2108.000000 [14:49:50] 2013/07/04 14:48 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1792.000000 [15:29:54] 2013/07/04 15:28 OK z-dat-s4-a /sql DISK OK - free space: /sql 67397 MB (11% inode=99%): [15:36:54] 2013/07/04 15:29 WARN z-dat-s4-a /sql DISK WARNING - free space: /sql 66990 MB (10% inode=99%): [18:42:06] 2013/07/04 18:35 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 1962.000000 [19:50:10] 2013/07/04 19:49 CRIT z-dat-s2-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3610.000000 [20:24:12] 2013/07/04 20:17 WARN wolfsbane / DISK WARNING - free space: / 6241 MB (20% inode=92%): [20:25:12] 2013/07/04 20:18 WARN wolfsbane /tmp DISK WARNING - free space: / 6242 MB (20% inode=92%): [21:11:16] 2013/07/04 21:10 CRIT wolfsbane / DISK CRITICAL - free space: / 3000 MB (10% inode=92%): [21:11:16] 2013/07/04 21:10 CRIT wolfsbane /tmp DISK CRITICAL - free space: / 3241 MB (10% inode=92%): [21:33:18] 2013/07/04 21:31 CRIT ortelius /tmp DISK CRITICAL - free space: /tmp 605 MB (2% inode=99%): [21:43:19] 2013/07/04 21:36 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2061 [22:04:23] known bug? -> /usr/bin/python: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory [22:19:21] 2013/07/04 22:18 OK z-dat-s2-b MySQL slave Uptime: 4024535 Threads: 10 Questions: 3697667217 Slow queries: 74228 Opens: 28976385 Flush tables: 1 Open tables: 256 Queries per second avg: 918.781 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1658 [23:03:26] 2013/07/04 23:02 WARN z-dat-s2-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3499.000000 [23:08:26] 2013/07/04 23:07 OK ortelius /tmp DISK OK - free space: /tmp 16402 MB (69% inode=99%): [23:22:28] 2013/07/04 23:21 OK z-dat-s2-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1783.000000