[00:03:02] 2013/11/03 00:02 WARN cassia MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3337 [00:03:02] 2013/11/03 00:02 WARN cassia s5 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3007.000000 [00:05:02] 2013/11/03 00:04 OK cassia MySQL slave Uptime: 12153520 Threads: 34 Questions: 7516602639 Slow queries: 215630 Opens: 27020 Flush tables: 1 Open tables: 14342 Queries per second avg: 618.471 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1643 [00:05:02] 2013/11/03 00:04 OK cassia s5 replag QUERY OK: 'SELECT ts_rc_age()' returned 1679.000000 [02:48:50] @replag [02:48:53] Danny_B: s1-rr-a: 2d 1h 31m 48s [+0.47 s/s]; s1-rr-a-wd: error; s1-user: 1d 8m 48s [-0.11 s/s]; s1-user-c: 2d 23h 13m 28s [+0.04 s/s]; s1-user-wd: 4w 3d 14h 10m 17s [+1.00 s/s]; s2-rr: 8h 49m 13s [-2.25 s/s]; s2-user: 8h 49m 13s [-2.25 s/s]; s2-user-c: error [02:48:54] Danny_B: s2-user-wd: 3w 6d 6h 28m 38s [+0.47 s/s]; s3-user: 20h 37m 51s [-0.60 s/s]; s3-user-wd: 16w 1d 8h 28m 3s [+1.00 s/s]; s4-rr-a: 15h 31m 1s [-1.32 s/s]; s4-user: 15h 30m 59s [-1.32 s/s]; s4-user-wd: 12w 13h 44m 33s [+1.00 s/s]; s5-rr-a: 17s [-3.48 s/s]; s5-rr-a-wd: 4w 8h 27m 50s [+0.50 s/s] [02:48:55] Danny_B: s5-user: 14h 30m 41s [-1.46 s/s]; s5-user-c: 21h 5m 1s [-0.54 s/s]; s6-user: 12h 23m 10s [-1.75 s/s]; s6-user-wd: 12w 13h 44m 34s [+1.00 s/s]; s7-user: 19h 10m 20s [-0.81 s/s]; s7-user-wd: 12w 13h 44m 35s [+1.00 s/s] [02:49:12] ow [04:19:15] 2013/11/03 04:18 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3100 [04:22:16] 2013/11/03 04:21 OK z-dat-s2-b MySQL slave Uptime: 4211544 Threads: 46 Questions: 1547125771 Slow queries: 28229 Opens: 14422677 Flush tables: 1 Open tables: 256 Queries per second avg: 367.353 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1653 [05:56:20] 2013/11/03 05:56 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3527.000000 [05:57:21] 2013/11/03 05:56 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3470 [06:00:20] 2013/11/03 06:00 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1604.000000 [06:01:20] 2013/11/03 06:00 OK z-dat-s5-b MySQL slave Uptime: 1444216 Threads: 14 Questions: 2021837468 Slow queries: 8362 Opens: 47521 Flush tables: 1 Open tables: 256 Queries per second avg: 1399.955 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1400 [06:35:24] 2013/11/03 06:28 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2044 [06:35:24] 2013/11/03 06:29 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2078.000000 [07:11:26] 2013/11/03 07:10 WARN z-dat-s4-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3388 [07:16:26] 2013/11/03 07:15 OK z-dat-s4-a MySQL slave Uptime: 5173816 Threads: 2 Questions: 322864934 Slow queries: 142 Opens: 497 Flush tables: 1 Open tables: 249 Queries per second avg: 62.403 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1770 [07:19:27] 2013/11/03 07:19 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3608.000000 [07:20:27] 2013/11/03 07:19 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3617 [07:37:29] 2013/11/03 07:36 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [07:37:29] 2013/11/03 07:37 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3474.000000 [07:40:28] 2013/11/03 07:39 WARN z-dat-s6-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3404 [07:47:29] 2013/11/03 07:46 OK z-dat-s6-a MySQL slave Uptime: 327917 Threads: 58 Questions: 35261892 Slow queries: 20692 Opens: 2000 Flush tables: 1 Open tables: 938 Queries per second avg: 107.532 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1715 [08:00:30] 2013/11/03 08:00 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1788.000000 [08:01:30] 2013/11/03 08:00 OK z-dat-s5-b MySQL slave Uptime: 1451416 Threads: 9 Questions: 2027272480 Slow queries: 8412 Opens: 47779 Flush tables: 1 Open tables: 256 Queries per second avg: 1396.754 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1747 [08:11:31] 2013/11/03 08:04 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2135 [08:11:32] 2013/11/03 08:05 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2172.000000 [08:40:32] 2013/11/03 08:40 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1728.000000 [08:41:32] 2013/11/03 08:40 OK z-dat-s5-b MySQL slave Uptime: 1453816 Threads: 9 Questions: 2028557613 Slow queries: 8442 Opens: 47949 Flush tables: 1 Open tables: 256 Queries per second avg: 1395.333 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1655 [08:59:34] 2013/11/03 08:53 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 2060.000000 [09:00:34] 2013/11/03 08:53 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2068 [09:39:35] 2013/11/03 09:39 WARN daphne MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [09:39:35] 2013/11/03 09:39 WARN daphne s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3548.000000 [09:53:36] 2013/11/03 09:53 CRIT z-dat-s5-b wikidata replag QUERY CRITICAL: 'SELECT ts_rc_age()' returned 3608.000000 [09:54:36] 2013/11/03 09:53 CRIT z-dat-s5-b MySQL slave SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3627 [09:58:36] 2013/11/03 09:58 OK daphne s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1777.000000 [09:59:36] 2013/11/03 09:59 OK daphne MySQL slave Uptime: 17158353 Threads: 104 Questions: 2536281232 Slow queries: 1726788 Opens: 500821 Flush tables: 1 Open tables: 2257 Queries per second avg: 147.816 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1643 [10:05:36] 2013/11/03 10:04 WARN z-dat-s5-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3569 [10:05:36] 2013/11/03 10:05 WARN z-dat-s5-b wikidata replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3464.000000 [10:19:36] 2013/11/03 10:19 OK z-dat-s5-b wikidata replag QUERY OK: 'SELECT ts_rc_age()' returned 1690.000000 [10:20:36] 2013/11/03 10:19 OK z-dat-s5-b MySQL slave Uptime: 1459756 Threads: 12 Questions: 2108464869 Slow queries: 8549 Opens: 48276 Flush tables: 1 Open tables: 256 Queries per second avg: 1444.395 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1596 [11:05:04] Federico Leva (Nemo) * Re: [Toolserver-l] perl: https-login not possible anymore [11:43:04] seth * Re: [Toolserver-l] perl: https-login not possible anymore [13:07:44] 2013/11/03 13:00 CRIT yarrow aliasd Connection refused [13:16:23] @replag [13:16:28] Danny_B: s1-rr-a: 2d 6h 26m 7s [+0.47 s/s]; s1-rr-a-wd: error; s1-user: 16h 39m 26s [-0.72 s/s]; s1-user-c: 3d 6h 43m 59s [+0.72 s/s]; s1-user-wd: 4w 4d 37m 50s [+1.00 s/s]; s2-rr: 30m 2s [-0.80 s/s]; s2-user: 30m 2s [-0.80 s/s]; s2-user-c: error [13:16:29] Danny_B: s2-user-wd: 3w 6d 3h 37m 16s [-0.27 s/s]; s3-user: 3h 47m 24s [-1.61 s/s]; s3-user-wd: 16w 1d 18h 55m 36s [+1.00 s/s]; s4-rr-a: 3m 24s [-1.48 s/s]; s4-user: 3m 25s [-1.48 s/s]; s4-user-wd: 12w 1d 12m 7s [+1.00 s/s]; s5-rr-a-wd: 4w 9h 49m 30s [+0.13 s/s]; s5-user-c: 8h 59m 22s [-1.16 s/s] [13:16:30] Danny_B: s6-user-wd: 12w 1d 12m 8s [+1.00 s/s]; s7-user: 1h 39m 5s [-1.68 s/s]; s7-user-wd: 12w 1d 12m 9s [+1.00 s/s] [13:23:46] 2013/11/03 13:17 WARN z-dat-s2-b MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2014 [13:30:46] 2013/11/03 13:30 OK z-dat-s2-b MySQL slave Uptime: 4244484 Threads: 27 Questions: 1576807426 Slow queries: 28576 Opens: 14533947 Flush tables: 1 Open tables: 256 Queries per second avg: 371.495 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780 [13:44:46] 2013/11/03 13:43 WARN z-dat-s7-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3596 [14:12:46] 2013/11/03 14:11 OK z-dat-s7-a MySQL slave Uptime: 2004165 Threads: 105 Questions: 301217877 Slow queries: 430340 Opens: 469055 Flush tables: 1 Open tables: 3480 Queries per second avg: 150.295 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1773 [15:28:51] 2013/11/03 15:28 WARN z-dat-s3-a MySQL slave SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3593 [15:28:51] 2013/11/03 15:28 WARN z-dat-s5-b s4 replag QUERY WARNING: 'SELECT ts_rc_age()' returned 3402.000000 [15:39:51] 2013/11/03 15:39 OK z-dat-s5-b s4 replag QUERY OK: 'SELECT ts_rc_age()' returned 1644.000000 [15:49:53] 2013/11/03 15:49 OK z-dat-s3-a MySQL slave Uptime: 356635 Threads: 55 Questions: 467235600 Slow queries: 74542 Opens: 1856476 Flush tables: 1 Open tables: 16384 Queries per second avg: 1310.122 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1790 [16:57:00] 2013/11/03 16:50 WARN ptolemy Load avg. WARNING - load average: 18.14, 17.20, 11.35 [16:59:59] 2013/11/03 16:59 OK ptolemy Load avg. OK - load average: 9.99, 14.32, 11.29 [22:00:19] 2013/11/03 21:54 WARN willow / DISK WARNING - free space: / 19390 MB (18% inode=98%): [22:00:19] 2013/11/03 21:53 WARN willow /tmp DISK WARNING - free space: / 19674 MB (18% inode=98%): [22:01:19] 2013/11/03 22:00 ?? willow Sun Grid Engine execd CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:02:19] 2013/11/03 22:01 WARN willow Sun Grid Engine execd NRPE: Unable to read output [22:13:20] 2013/11/03 22:12 CRIT amaranth PING PING CRITICAL - Packet loss = 100% [22:13:20] 2013/11/03 22:12 CRIT amaranth SMTP CRITICAL - Socket timeout after 10 seconds [22:13:21] 2013/11/03 22:12 CRIT web.amaranth PING PING CRITICAL - Packet loss = 100% [22:13:21] 2013/11/03 22:12 CRIT web.amaranth jira.toolserver.org CRITICAL - Socket timeout after 16 seconds [22:13:21] 2013/11/03 22:12 CRIT web.amaranth wiki.toolserver.org CRITICAL - Socket timeout after 16 seconds [22:14:20] 2013/11/03 22:13 OK amaranth PING PING OK - Packet loss = 0%, RTA = 127.94 ms [22:14:20] 2013/11/03 22:13 OK amaranth SMTP SMTP OK - 0.517 sec. response time [22:14:21] 2013/11/03 22:13 OK web.amaranth PING PING OK - Packet loss = 0%, RTA = 125.80 ms [22:14:21] 2013/11/03 22:13 OK web.amaranth jira.toolserver.org HTTP OK: HTTP/1.1 200 OK - 522 bytes in 0.563 second response time [22:14:21] 2013/11/03 22:13 OK web.amaranth wiki.toolserver.org HTTP OK: HTTP/1.1 200 OK - 349 bytes in 0.734 second response time [22:16:21] 2013/11/03 22:16 OK willow / DISK OK - free space: / 32555 MB (31% inode=99%): [22:16:21] 2013/11/03 22:15 OK willow /tmp DISK OK - free space: / 32715 MB (31% inode=99%):