[00:00:53] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [00:00:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 730173.000000 [00:01:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [00:03:44] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [00:06:33] Load avg. on willow is WARNING: WARNING - load average: 13.74, 18.05, 15.21 [00:11:33] Load avg. on willow is OK: OK - load average: 13.70, 14.95, 14.53 [00:13:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:13:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:13:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99370 MB (10% inode=98%): [00:13:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99369 MB (10% inode=98%): [00:43:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14438.000000 [00:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 731903 [00:45:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 471759 MB (8% inode=43%): [00:49:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:49:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:49:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:53:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10828.000000 [00:54:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:54] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10941.000000 [00:55:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99334 MB (10% inode=98%): [00:59:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [01:00:54] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [01:00:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 732478.000000 [01:01:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [01:04:14] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [01:18:33] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:23:13] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:43:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18037.000000 [01:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 734081 [01:45:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:46:37] @replag [01:46:38] Dispenser: s1-rr-a: 5h 4m 22s [+0.29 s/s]; s1-user: 1w 1d 11h 55m 12s [+0.58 s/s]; s2-user: 1m 5s [+0.00 s/s]; s3-rr-a: 2m 29s [+0.00 s/s]; s3-user: 2m 29s [+0.00 s/s]; s4-rr-a: 3h 54m 6s [+0.22 s/s]; s4-user: 3h 54m 6s [+0.22 s/s]; s6-rr-a: 3h 52m 40s [+0.30 s/s] [01:46:39] Dispenser: s6-user: 3h 52m 40s [+0.30 s/s]; s7-rr-a: 3h 49m 40s [+0.22 s/s]; s7-user: 3h 49m 40s [+0.22 s/s] [01:48:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 471662 MB (8% inode=43%): [01:49:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:49:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:49:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:53:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14429.000000 [01:56:03] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14602.000000 [01:59:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [02:00:54] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [02:00:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 734533.000000 [02:01:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [02:04:14] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [02:43:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21637.000000 [02:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 736294 [02:45:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:46:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43420 MB (10% inode=99%): [02:49:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:49:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:49:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:49:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469664 MB (8% inode=43%): [02:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:51:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 51267 MB (12% inode=99%): [02:53:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18028.000000 [02:56:03] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18202.000000 [02:59:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:01:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [03:01:54] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [03:01:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 736823.000000 [03:04:14] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [03:13:14] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 48620 MB (5% inode=99%): [03:43:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25238.000000 [03:45:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 738379 [03:45:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:49:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:49:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469382 MB (8% inode=42%): [03:50:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:50:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:51:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:54:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21688.000000 [03:56:04] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21801.000000 [03:59:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:01:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [04:01:53] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [04:01:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739008.000000 [04:04:14] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [04:09:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43975 MB (10% inode=99%): [04:43:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28838.000000 [04:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 740899 [04:46:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:47:02] [[Special:Log/newusers]] create 10 * Klauskettner * (New user account) [04:49:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469523 MB (8% inode=42%): [04:50:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:50:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:50:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:51:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:54:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25288.000000 [04:57:03] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25462.000000 [04:57:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 131979 MB (13% inode=99%): [04:59:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:01:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [05:01:54] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [05:01:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741555.000000 [05:03:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24366 MB (5% inode=99%): [05:05:14] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [05:27:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99108 MB (10% inode=98%): [05:27:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99108 MB (10% inode=98%): [05:42:23] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 41075 MB (3% inode=98%): [05:43:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32438.000000 [05:45:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 742981 [05:46:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:49:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469412 MB (8% inode=42%): [05:50:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:50:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:51:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:54:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28888.000000 [05:57:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29082.000000 [06:00:03] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:01:33] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [06:01:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743427.000000 [06:02:03] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [06:03:17] I'm waiting on a job submitted over two hours ago on willow to exit the queue and start running. [06:03:36] Its memory usage is relatively low (virtual_free=100M) and its settings haven't changed in a while. [06:03:46] Anyone know if there's a problem I should be aware of? [06:05:23] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [06:06:24] Sigh. [06:06:39] This channel is utterly useless as any messages get drowned out by tsnag. [06:10:56] [[Cron]] ! 10https://wiki.toolserver.org/w/index.php?diff=7466&oldid=7309&rcid=10170 * 93.50.129.38 * (-1) (/* crontab syntax */ typo) [06:44:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36098.000000 [06:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 744823 [06:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:50:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:50:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:50:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469319 MB (8% inode=42%): [06:51:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:55:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27231.000000 [06:55:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:23] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:57:43] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [06:57:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 98857 MB (10% inode=98%): [06:57:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 98755 MB (10% inode=98%): [06:59:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29301 MB (7% inode=99%): [07:00:14] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30705 [07:01:34] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20920 [07:01:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 745247.000000 [07:02:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31610 [07:02:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24289 MB (5% inode=99%): [07:05:33] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [07:11:03] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3192.000000 [07:11:33] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3099 [07:12:33] MySQL slave on daphne is OK: Uptime: 12547037 Threads: 17 Questions: 909555628 Slow queries: 416198 Opens: 1300 Flush tables: 2 Open tables: 349 Queries per second avg: 72.491 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1377 [07:13:03] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 621.000000 [07:39:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35886 MB (8% inode=99%): [07:44:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39698.000000 [07:45:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 746709 [07:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:50:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:50:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:50:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469187 MB (8% inode=42%): [07:51:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:57:53] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31301.000000 [08:00:14] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:00:43] @@replag [08:00:45] er [08:00:47] @replag [08:00:47] cybug: s1-rr-a: 11h 18m 32s [+1.00 s/s]; s1-user: 1w 1d 15h 32m 53s [+0.52 s/s]; s2-user: 13s [-0.01 s/s]; s4-user: 8h 44m 45s [-0.12 s/s]; s6-rr-a: 8h 37m 1s [-0.21 s/s]; s6-user: 8h 37m 1s [-0.21 s/s]; s7-rr-a: 10h 3m 20s [+0.99 s/s]; s7-user: 10h 3m 20s [+0.99 s/s] [08:00:53] blur: ^ [08:01:13] oh [08:01:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 747209.000000 [08:02:33] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [08:05:34] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [08:05:51] @replag [08:05:52] cybug: s1-rr-a: 11h 23m 36s [+1.00 s/s]; s1-user: 1w 1d 15h 35m 50s [+0.58 s/s]; s3-rr-a: 15s [+0.00 s/s]; s3-user: 15s [+0.00 s/s]; s4-user: 8h 49m 49s [+1.00 s/s]; s6-rr-a: 8h 42m 5s [+1.00 s/s]; s6-user: 8h 42m 5s [+1.00 s/s]; s7-rr-a: 10h 8m 24s [+1.00 s/s] [08:05:53] cybug: s7-user: 10h 8m 24s [+1.00 s/s] [08:06:04] +1.00 s/s [08:06:09] many are [08:06:13] what on earth is going on? [08:19:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 48559 MB (11% inode=99%): [08:26:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 28689 MB (7% inode=99%): [08:29:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22092 MB (5% inode=99%): [08:44:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43297.000000 [08:45:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 749001 [08:47:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:48:33] Load avg. on willow is WARNING: WARNING - load average: 10.70, 18.30, 15.59 [08:48:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29612 MB (7% inode=99%): [08:49:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23862 MB (5% inode=99%): [08:51:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:51:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:51:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:51:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469062 MB (8% inode=42%): [08:52:33] Load avg. on willow is OK: OK - load average: 8.56, 13.87, 14.43 [08:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:57:53] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34900.000000 [09:01:13] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:01:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 749673.000000 [09:02:33] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [09:06:33] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [09:09:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 30270 MB (7% inode=99%): [09:10:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24032 MB (5% inode=99%): [09:24:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 52340 MB (12% inode=99%): [09:45:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46957.000000 [09:45:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 751517 [09:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:51:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 468883 MB (8% inode=42%): [09:52:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:52:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:58:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38521.000000 [10:01:23] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:01:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 752354.000000 [10:03:03] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [10:07:03] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [10:21:43] Load avg. on adenia is WARNING: WARNING - load average: 17.30, 15.20, 9.30 [10:22:43] Load avg. on adenia is OK: OK - load average: 13.75, 14.59, 9.46 [10:45:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50558.000000 [10:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 754150 [10:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:51:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 468728 MB (8% inode=42%): [10:52:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:52:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:59:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42180.000000 [11:01:23] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:02:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 754774.000000 [11:03:03] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [11:07:03] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [11:45:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54157.000000 [11:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 756717 [11:47:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 468567 MB (8% inode=42%): [11:52:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:52:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:52:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:59:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45800.000000 [12:01:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:02:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 757536.000000 [12:03:03] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [12:07:33] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [12:19:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39982 MB (9% inode=99%): [12:38:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23619 MB (5% inode=99%): [12:41:39] [[Special:Log/newusers]] create 10 * Schrodinger's cat is alive * (New user account) [12:45:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57758.000000 [12:45:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 759439 [12:47:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 468400 MB (8% inode=42%): [12:51:51] @replag [12:51:51] Dispenser: s1-rr-a: 16h 9m 36s [+1.00 s/s]; s1-user: 1w 1d 19h 1m 47s [+0.72 s/s]; s3-rr-a: 1m 51s [+0.01 s/s]; s3-user: 1m 51s [+0.01 s/s]; s4-user: 13h 35m 49s [+1.00 s/s]; s6-rr-a: 13h 28m 5s [+1.00 s/s]; s6-user: 13h 28m 5s [+1.00 s/s]; s7-rr-a: 14h 54m 24s [+1.00 s/s] [12:51:52] Dispenser: s7-user: 14h 54m 24s [+1.00 s/s] [12:52:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:52:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:52:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:52:31] What the happened to rosemary? [12:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:55:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36781 MB (9% inode=99%): [12:59:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49401.000000 [13:01:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:02:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 760115.000000 [13:03:03] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [13:07:33] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [13:12:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24121 MB (5% inode=99%): [13:24:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 44963 MB (11% inode=99%): [13:25:15] hello all [13:31:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36895 MB (9% inode=99%): [13:38:33] hi DaB! It seems the cluster's stopped replicating [13:38:37] @replag [13:38:38] Dispenser: s1-rr-a: 16h 56m 22s [+1.00 s/s]; s1-user: 1w 1d 19h 37m 39s [+0.74 s/s]; s3-rr-a: 50s [-0.27 s/s]; s3-user: 50s [-0.27 s/s]; s4-user: 14h 22m 35s [+1.00 s/s]; s6-rr-a: 14h 14m 51s [+1.00 s/s]; s6-user: 14h 14m 51s [+1.00 s/s]; s7-rr-a: 15h 41m 10s [+1.00 s/s] [13:38:39] Dispenser: s7-user: 15h 41m 10s [+1.00 s/s] [13:40:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23259 MB (5% inode=99%): [13:45:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61358.000000 [13:45:41] Dispenser: https://jira.toolserver.org/browse/MNT-1258 – I'm looking into it [13:45:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 762199 [13:47:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 468254 MB (8% inode=42%): [13:52:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61723 [13:52:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:52:21] @replag [13:52:21] DaBPunkt: s1-rr-a: 17h 8m 54s [+0.91 s/s]; s1-user: 1w 1d 19h 47m 59s [+0.75 s/s]; s3-rr-a: 27s [-0.03 s/s]; s3-user: 27s [-0.03 s/s]; s4-user: 14h 36m 18s [+1.00 s/s]; s6-rr-a: 14h 28m 34s [+1.00 s/s]; s6-user: 14h 28m 34s [+1.00 s/s]; s7-rr-a: 15h 54m 53s [+1.00 s/s] [13:52:22] DaBPunkt: s7-user: 15h 54m 53s [+1.00 s/s] [13:52:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:52:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:55:02] @replag [13:55:02] DaBPunkt: s1-rr-a: 17h 11m 8s [+0.83 s/s]; s1-user: 1w 1d 19h 49m 39s [+0.62 s/s]; s3-rr-a: 45s [+0.11 s/s]; s3-user: 45s [+0.11 s/s]; s4-user: 14h 38m 59s [+1.00 s/s]; s6-rr-a: 14h 31m 15s [+1.00 s/s]; s6-user: 14h 31m 15s [+1.00 s/s]; s7-rr-a: 15h 57m 34s [+1.00 s/s] [13:55:03] DaBPunkt: s7-user: 15h 57m 34s [+1.00 s/s] [13:59:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50635.000000 [13:59:43] @replag [13:59:43] DaBPunkt: s1-rr-a: 17h 15m 49s [+1.00 s/s]; s1-user: 1w 1d 19h 52m 44s [+0.66 s/s]; s2-user: 10s [+0.01 s/s]; s4-user: 13h 59m 58s [-8.33 s/s]; s6-rr-a: 14h 16m 37s [-3.12 s/s]; s6-user: 14h 16m 37s [-3.12 s/s]; s7-rr-a: 16h 2m 2s [+0.95 s/s]; s7-user: 16h 2m 3s [+0.96 s/s] [14:01:34] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50404 [14:02:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 762843.000000 [14:03:03] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48737 [14:03:40] @replag [14:03:40] DaBPunkt: s1-rr-a: 17h 18m 53s [+0.78 s/s]; s1-user: 1w 1d 19h 55m 10s [+0.62 s/s]; s3-rr-a: 20s [-0.05 s/s]; s3-user: 20s [-0.05 s/s]; s4-user: 13h 53m 30s [-1.64 s/s]; s6-rr-a: 13h 24m 10s [-13.28 s/s]; s6-user: 13h 24m 10s [-13.28 s/s]; s7-rr-a: 16h 5m 49s [+0.96 s/s] [14:03:41] DaBPunkt: s7-user: 16h 5m 49s [+0.95 s/s] [14:07:33] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57940 [14:12:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29149 MB (7% inode=99%): [14:20:52] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24201 MB (5% inode=99%): [14:38:50] DaBPunkt: what do you think of TS-1463? [14:38:54] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49245 MB (12% inode=99%): [14:39:27] giftpflanze: I work on it [14:39:33] :) [14:39:57] giftpflanze: "Portal-Sichtungsbot" do you have a bo-name? [14:40:01] bot [14:40:14] everything linkfa-bot [14:40:21] -everything [14:45:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62179.000000 [14:45:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 764736 [14:47:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:47:52] @replag [14:47:52] DaBPunkt: s1-rr-a: 17h 12m 15s [-0.88 s/s]; s1-user: 1w 1d 20h 27m 26s [+0.68 s/s]; s4-user: 6h 12m 25s [-8.11 s/s]; s6-rr-a: 3h 24m 13s [-8.09 s/s]; s6-user: 3h 24m 13s [-8.09 s/s]; s7-rr-a: 12h 49m 8s [-5.69 s/s]; s7-user: 12h 49m 8s [-5.69 s/s] [14:51:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467983 MB (8% inode=42%): [14:52:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61552 [14:52:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:52:27] giftpflanze: resolved [14:52:38] oi [14:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:53:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:53:59] DaBPunkt: the password files didn't contain anything but passwords, usernames and such? [14:54:41] i hope i can easily reconstruct the format for me [14:56:24] giftpflanze: I will create you a dummy [14:56:48] thx [14:57:43] done [14:58:18] it's root-only [14:58:55] fixed [14:59:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15905.000000 [15:00:21] DaBPunkt: are there no gvmbot sources? [15:00:57] no [15:01:33] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14608 [15:02:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 765484.000000 [15:02:32] hm [15:04:04] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9001 [15:07:33] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35989 [15:16:24] !ops Fluff|away, you can't get rid of the problem like this, you have your own personal hygiene to worry about. [15:18:17] DaBPunkt: was there a cron(ie)tab? [15:19:03] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3564 [15:22:33] giftpflanze: not as far as I see. I have a cronjobs dir in his home, but it is empty [15:22:40] I → he [15:24:33] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3401 [15:24:33] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3401.000000 [15:25:02] MySQL slave on z-dat-s6-a is OK: Uptime: 3440871 Threads: 7 Questions: 766491632 Slow queries: 239215 Opens: 7155504 Flush tables: 1 Open tables: 3075 Queries per second avg: 222.760 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1471 [15:26:10] did you look on the hosts? (submit …) [15:28:33] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1709.000000 [15:28:34] MySQL slave on z-dat-s4-a is OK: Uptime: 5420019 Threads: 6 Questions: 309857331 Slow queries: 55053 Opens: 125685 Flush tables: 1 Open tables: 1110 Queries per second avg: 57.169 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1710 [15:31:56] giftpflanze: yes [15:32:28] meh, ok [15:37:30] DaBPunkt: that dummy, what was its filename and location? [15:38:33] ./credits/LinkFA [15:38:39] o.O [15:45:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58930.000000 [15:46:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 767304 [15:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467860 MB (8% inode=42%): [15:52:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58150 [15:52:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:53:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:02:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 767881.000000 [16:07:33] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15700 [16:45:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54737.000000 [16:46:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 769747 [16:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467783 MB (8% inode=42%): [16:52:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54216 [16:52:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:52:45] @replag [16:52:45] Dispenser: s1-rr-a: 15h 1m 15s [-1.51 s/s]; s1-user: 1w 1d 21h 52m 47s [+0.60 s/s]; s3-rr-a: 1m 28s [+0.15 s/s]; s3-user: 1m 28s [+0.15 s/s]; s7-rr-a: 1h 4m 50s [-5.35 s/s]; s7-user: 1h 4m 50s [-5.35 s/s] [16:52:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:54:33] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3264 [16:57:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 114305 MB (11% inode=99%): [16:57:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:59:33] MySQL slave on z-dat-s7-a is OK: Uptime: 5403848 Threads: 4 Questions: 1586136156 Slow queries: 191267 Opens: 13084630 Flush tables: 1 Open tables: 6959 Queries per second avg: 293.519 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1746 [17:02:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 770316.000000 [17:16:51] @replag [17:16:51] Earwig: s1-rr-a: 14h 25m 44s [-1.47 s/s]; s1-user: 1w 1d 22h 7m 58s [+0.63 s/s] [17:39:07] @replag [17:39:07] Earwig: s1-rr-a: 13h 56m 12s [-1.30 s/s]; s1-user: 1w 1d 22h 20m 6s [+0.58 s/s]; s3-rr-a: 34s [-0.05 s/s]; s3-user: 34s [-0.05 s/s] [17:43:24] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 43909 MB (3% inode=98%): [17:46:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49929.000000 [17:46:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 771903 [17:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467681 MB (8% inode=42%): [17:52:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49571 [17:52:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:53:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:57:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:02:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 772409.000000 [18:39:57] @replag [18:39:58] russblau: s1-rr-a: 12h 54m 56s [-1.01 s/s]; s1-user: 1w 1d 22h 55m 31s [+0.58 s/s]; s2-user: 18s [-]; s3-rr-a: 1m 9s [+0.01 s/s]; s3-user: 1m 9s [+0.01 s/s] [18:46:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46545.000000 [18:46:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 774016 [18:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467617 MB (8% inode=42%): [18:52:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46192 [18:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:53:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:54:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:57:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:57:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 99799 MB (10% inode=98%): [18:57:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 99799 MB (10% inode=98%): [18:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:03:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 774572.000000 [19:46:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42394.000000 [19:46:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 776117 [19:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:51:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467579 MB (8% inode=42%): [19:52:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42052 [19:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:53:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:54:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:58:44] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:59:52] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:03:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 776684.000000 [20:04:40] @replag [20:04:40] russblau: s1-rr-a: 11h 27m 33s [-1.03 s/s]; s1-user: 1w 1d 23h 45m 38s [+0.59 s/s]; s3-rr-a: 19s [-0.01 s/s]; s3-user: 19s [-0.01 s/s] [20:24:13] / on hemlock is WARNING: DISK WARNING - free space: / 4193 MB (20% inode=89%): [20:29:49] dschwen: Do you get my Toolserver emails? [20:29:58] huh? [20:30:04] Hi Dispenser [20:30:13] I don't think so [20:30:15] when? [20:30:39] I sent an email asking where you keep the label server stuff so I could look into possibly improving or speeding it up [20:30:40] odd, I do have a mail forward set [20:30:54] oh [20:31:16] the labelserver now is just a small php script and all data is stored in mysql [20:31:29] by the way, brand new: [20:31:41] http://toolserver.org/~dschwen/wma/index_dev.html [20:31:46] zoom in! [20:32:10] I've been looking at it. Nice to have commons images on the moon. [20:32:40] yeah, but zoom in [20:32:45] on the base map [20:32:47] Now if we could get the WMF to invest in a drone program... [20:33:00] yeah, but... [20:33:06] the base map, zoom in! [20:35:22] BTW, increase the tile size when you get around to changing the projection [20:36:02] yeah yeah [20:36:13] so... [20:36:20] client-side rendered tiles [20:36:24] boom [20:46:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39506.000000 [20:46:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 778308 [20:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467512 MB (8% inode=42%): [20:52:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39274 [20:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:53:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:54:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:58:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:04:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 778901.000000 [21:17:56] hello [21:18:22] is replication lag the cause of the geobox template at the top of this article taking some time to update? http://en.wikipedia.org/wiki/List_of_highest_paved_roads_in_Europe_by_country [21:18:45] "Map of all coordinates from Google"<-- this option [21:24:14] / on hemlock is WARNING: DISK WARNING - free space: / 4190 MB (20% inode=89%): [21:27:24] mrpotatoe, that shouldn't really have anything to do with replag; the URL does point to toolserver.org but it is just being redirected to Google, it doesn't seem to access the database replicas at all. I suppose you'd have to look at the source code for the tool to be sure of that, though, and I haven't done that. [21:27:29] looks fast enough to me [21:28:06] ok thanks russblau [21:28:22] Nor can you read the source since para's removed read permissions [21:28:28] it only takes a few hours to update (maybe a day) so it's not a big issue [21:46:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36354.000000 [21:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 780666 [21:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:51:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467459 MB (8% inode=42%): [21:52:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36135 [21:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:54:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:54:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:58:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:04:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 781278.000000 [22:24:14] / on hemlock is WARNING: DISK WARNING - free space: / 4188 MB (20% inode=89%): [22:46:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33679.000000 [22:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 783072 [22:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:51:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467428 MB (8% inode=42%): [22:52:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:53:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33389 [22:54:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:54:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:58:43] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:59:54] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:04:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 783731.000000 [23:24:13] / on hemlock is WARNING: DISK WARNING - free space: / 4186 MB (20% inode=89%): [23:46:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31320.000000 [23:47:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 785626 [23:47:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:52:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 466041 MB (8% inode=42%): [23:52:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:53:04] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31143 [23:54:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:54:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:58:44] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:59:53] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates).