[00:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 92576 MB (9% inode=99%): [00:20:15] a simple one-field update on the main user database server is taking forever... [00:21:03] a select for the same field comes back instantly (using the same where clause) [00:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1214836 [00:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:38:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1214934.000000 [00:40:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:48:29] stwalkerster: enwiki currently has a lot of write activity from the rev_hash inserts [00:51:09] Dispenser: wrong cluster AFAIK, but I discovered a query had a write lock on the entire DB, so the problem is resolved :) [01:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1216818 [01:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:38:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1216923.000000 [01:40:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1218508 [02:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:38:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1218611.000000 [02:40:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:10:31] rev_sha1 [03:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1220389 [03:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:38:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1220502.000000 [03:40:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:30:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1222266 [04:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:38:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1222359.000000 [04:40:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:50:53] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1224029 [05:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:38:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1224120.000000 [05:40:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 32008 MB (2% inode=98%): [06:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1226127 [06:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:38:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1226239.000000 [06:40:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1228139 [07:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:36:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:36:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:36:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:38:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1228240.000000 [07:39:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42833 MB (10% inode=99%): [07:39:03] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 94267 MB (9% inode=98%): [07:39:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 94262 MB (9% inode=98%): [07:40:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:55:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42818 MB (10% inode=99%): [07:55:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 94127 MB (9% inode=98%): [08:01:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:01:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:20:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42786 MB (10% inode=99%): [08:20:44] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 94165 MB (9% inode=98%): [08:20:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 94165 MB (9% inode=98%): [08:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1230006 [08:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:38:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1230108.000000 [08:40:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:41:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:54:12] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:54:14] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:54:14] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42757 MB (10% inode=99%): [08:55:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93475 MB (9% inode=98%): [08:55:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93475 MB (9% inode=98%): [09:10:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:14] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:29:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42659 MB (10% inode=99%): [09:29:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93119 MB (9% inode=98%): [09:29:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93119 MB (9% inode=98%): [09:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1231930 [09:35:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:38:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1232053.000000 [09:40:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1234340 [10:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:39:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1234468.000000 [10:40:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1236398 [11:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:38:10] hello [11:38:24] i have someone maintaining hardware in the datacenter [11:38:28] so we repair wolfsbane [11:38:36] will be back in some minutes [11:39:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1236509.000000 [11:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:44:43] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:55:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92283 MB (9% inode=98%): [11:55:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42512 MB (10% inode=99%): [11:55:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92283 MB (9% inode=98%): [12:09:12] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:12:32] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.022 second response time [12:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 92695 MB (9% inode=99%): [12:19:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:24:52] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:52] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:24:52] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:12] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:13] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:13] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:13] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:13] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:13] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:13] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:14] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:14] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:15] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:15] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:16] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:16] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:17] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:33] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:33] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:33] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:52] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:53] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:53] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:53] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:26:32] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [12:26:42] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [12:26:52] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [12:26:53] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [12:26:53] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [12:26:53] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [12:26:53] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [12:27:03] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [12:27:03] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [12:30:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:30:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:30:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:30:33] MySQL on z-dat-s4-a is OK: Uptime: 6359747 Threads: 17 Questions: 419943070 Slow queries: 81605 Opens: 143553 Flush tables: 1 Open tables: 1126 Queries per second avg: 66.31 [12:30:33] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 55461 MB (13% inode=99%): [12:30:43] MySQL slave on z-dat-s4-a is OK: Uptime: 6359747 Threads: 17 Questions: 419943070 Slow queries: 81605 Opens: 143553 Flush tables: 1 Open tables: 1126 Queries per second avg: 66.31 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 733 [12:30:43] MySQL on z-dat-s6-a is OK: Uptime: 4380811 Threads: 17 Questions: 957940001 Slow queries: 298256 Opens: 9262675 Flush tables: 1 Open tables: 3108 Queries per second avg: 218.667 [12:30:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93374 MB (9% inode=98%): [12:30:43] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1736 MB (99% inode=99%): [12:30:43] Load avg. on z-dat-s6-a is OK: OK - load average: 0.02, 0.28, 1.28 [12:30:43] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1738 MB (99% inode=99%): [12:30:44] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93374 MB (9% inode=98%): [12:30:44] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1737 MB (99% inode=99%): [12:30:44] / on z-dat-s3-a is OK: DISK OK - free space: / 8055 MB (26% inode=85%): [12:30:45] Load avg. on z-dat-s4-a is OK: OK - load average: 0.02, 0.28, 1.28 [12:30:46] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35498 MB (8% inode=99%): [12:30:46] Load avg. on z-dat-s7-a is OK: OK - load average: 0.02, 0.28, 1.28 [12:31:03] SMTP on z-dat-s7-a is OK: SMTP OK - 0.271 sec. response time [12:31:03] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:31:03] SMTP on z-dat-s6-a is OK: SMTP OK - 0.123 sec. response time [12:31:03] SMTP on z-dat-s4-a is OK: SMTP OK - 0.191 sec. response time [12:31:03] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:31:03] / on hyacinth is OK: DISK OK - free space: / 8055 MB (26% inode=85%): [12:31:03] /tmp on hyacinth is OK: DISK OK - free space: /tmp 1729 MB (100% inode=99%): [12:31:13] SMTP on z-dat-s3-a is OK: SMTP OK - 0.201 sec. response time [12:31:13] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:31:13] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:31:22] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:31:32] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [12:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1238709 [12:35:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:39:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1238843.000000 [12:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:51:32] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2010 [12:57:12] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:57:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:57:33] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [12:58:42] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2357 [12:58:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43282 MB (10% inode=99%): [12:58:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93356 MB (9% inode=98%): [13:08:52] MySQL slave on z-dat-s3-a is OK: Uptime: 4431288 Threads: 16 Questions: 5057325753 Slow queries: 299560 Opens: 50055592 Flush tables: 1 Open tables: 16384 Queries per second avg: 1141.276 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1751 [13:12:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:12:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:12:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:12:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93398 MB (9% inode=98%): [13:19:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:24:23] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [13:24:43] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:24:53] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [13:24:53] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [13:24:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:24:53] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [13:25:12] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [13:31:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43270 MB (10% inode=99%): [13:35:13] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [13:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:39:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240685.000000 [13:40:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93429 MB (9% inode=98%): [13:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:48:12] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2170.000000 [13:48:22] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2181.000000 [13:48:43] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2192.000000 [13:49:03] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2213.000000 [13:49:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2215.000000 [13:49:03] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2216.000000 [18:02:19] Marlen Caemmerer * Re: [Toolserver-l] Replag [18:03:43] MySQL slave on rosemary is OK: Uptime: 10032373 Threads: 23 Questions: 4003730813 Slow queries: 724190 Opens: 16172 Flush tables: 6 Open tables: 734 Queries per second avg: 399.81 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [18:09:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:09:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:09:53] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3320.000000 [18:10:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:10:23] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2700 [18:10:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16238 [18:11:23] MySQL slave on daphne is OK: Uptime: 13536962 Threads: 19 Questions: 990399445 Slow queries: 447700 Opens: 1300 Flush tables: 2 Open tables: 349 Queries per second avg: 73.162 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1467 [18:11:52] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 837.000000 [18:12:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11078.000000 [18:12:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17226.000000 [18:13:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9296.000000 [18:13:04] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6540.000000 [18:13:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8749.000000 [18:17:03] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3523.000000 [18:17:23] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3300 [18:21:03] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1714.000000 [18:21:22] MySQL slave on cassia is OK: Uptime: 8231456 Threads: 24 Questions: 10648423498 Slow queries: 2204632 Opens: 23428062 Flush tables: 1 Open tables: 16351 Queries per second avg: 1293.625 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1476 [18:24:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:25:13] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6650 [18:25:23] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6126 [18:25:33] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8929 [18:25:43] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4016 [18:27:43] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3342 [18:27:50] @replag [18:27:50] russblau: s1-rr-a: 3h 36m 4s [-3.66 s/s]; s1-user: 2w 12h 4m 19s [+0.74 s/s]; s2-user: 2h 56m 42s [-5.51 s/s]; s2-user-c: 1h 39m 9s [-4.95 s/s]; s3-rr-a: 53m 34s [-7.05 s/s]; s3-user: 53m 34s [-7.05 s/s]; s4-user: 1h 43m 8s [-4.05 s/s]; s5-user-c: 1h 39m 9s [-4.95 s/s] [18:27:51] russblau: s6-rr-a: 1h 26m 43s [-6.54 s/s]; s6-user: 1h 26m 43s [-6.54 s/s]; s7-rr-a: 2h 20m 25s [-2.97 s/s]; s7-user: 2h 20m 25s [-2.97 s/s] [18:33:13] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3425.000000 [18:33:43] MySQL slave on z-dat-s3-a is OK: Uptime: 4450778 Threads: 22 Questions: 5073511087 Slow queries: 300055 Opens: 50342012 Flush tables: 1 Open tables: 16384 Queries per second avg: 1139.915 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1411 [18:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1253371 [18:35:22] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3401 [18:37:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3181.000000 [18:38:13] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1538.000000 [18:39:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1253546.000000 [18:39:56] hello allo [18:40:00] @replag [18:40:00] DaBPunkt: s1-rr-a: 3h 21m 58s [-1.17 s/s]; s1-user: 2w 12h 12m 58s [+0.66 s/s]; s2-user: 2h 34m 40s [-2.41 s/s]; s2-user-c: 33m 37s [-6.54 s/s]; s4-user: 1h 2m 42s [-2.80 s/s]; s5-user-c: 33m 37s [-6.54 s/s]; s6-rr-a: 38m 30s [-3.07 s/s]; s6-user: 38m 30s [-3.07 s/s] [18:40:01] DaBPunkt: s7-rr-a: 1h 46m 2s [-2.58 s/s]; s7-user: 1h 46m 2s [-2.58 s/s] [18:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:41:02] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1800.000000 [18:41:12] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3493 [18:41:33] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3494.000000 [18:44:22] MySQL slave on z-dat-s6-a is OK: Uptime: 4403230 Threads: 9 Questions: 960155044 Slow queries: 299274 Opens: 9266705 Flush tables: 1 Open tables: 3099 Queries per second avg: 218.56 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1738 [18:53:33] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [18:54:13] MySQL slave on z-dat-s4-a is OK: Uptime: 6382759 Threads: 18 Questions: 421469701 Slow queries: 81980 Opens: 143741 Flush tables: 1 Open tables: 1130 Queries per second avg: 66.32 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1592 [19:01:33] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3548 [19:09:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:09:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:10:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:10:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10510 [19:12:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10353.000000 [19:13:32] MySQL slave on z-dat-s7-a is OK: Uptime: 6362286 Threads: 11 Questions: 1780436542 Slow queries: 222016 Opens: 14966512 Flush tables: 1 Open tables: 6993 Queries per second avg: 279.842 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1591 [19:24:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1255956 [19:39:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1256129.000000 [19:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:08:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:08:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:08:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:09:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:09:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:10:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:10:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6727 [20:10:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93086 MB (9% inode=98%): [20:10:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42791 MB (10% inode=99%): [20:10:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93086 MB (9% inode=98%): [20:12:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6646.000000 [20:15:12] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:15:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:24:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:29:19] @replag [20:29:19] jeremyb: s1-rr-a: 1h 31m 37s [-0.83 s/s]; s1-user: 2w 13h 30m 12s [+0.70 s/s]; s3-rr-a: 32s [-0.01 s/s]; s3-user: 32s [-0.01 s/s] [20:33:05] Merlijn van Deen * [Toolserver-l] ts-reports back online with improved caching [20:34:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92208 MB (9% inode=98%): [20:34:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42769 MB (10% inode=99%): [20:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1258462 [20:39:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1258641.000000 [20:39:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:39:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:43:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:43:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:43:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:43:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92231 MB (9% inode=98%): [20:43:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92231 MB (9% inode=98%): [20:43:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42764 MB (10% inode=99%): [20:50:58] DaBPunkt: any ideia fro when apache will be installed? [20:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:56:42] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3542 [20:56:43] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3547.000000 [21:10:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:11:05] Alchimista: I work on it [21:13:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92009 MB (9% inode=98%): [21:13:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42730 MB (10% inode=99%): [21:13:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92009 MB (9% inode=98%): [21:18:43] MySQL slave on rosemary is OK: Uptime: 10044073 Threads: 23 Questions: 4009890417 Slow queries: 725109 Opens: 16176 Flush tables: 6 Open tables: 734 Queries per second avg: 399.229 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1746 [21:18:43] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1749.000000 [21:24:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:31:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:31:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:31:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:31:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91620 MB (9% inode=98%): [21:31:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42704 MB (10% inode=99%): [21:31:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91620 MB (9% inode=98%): [21:35:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1261605 [21:39:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1261794.000000 [21:39:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:39:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:53:32] nacht ts [21:55:41] @replag [21:55:42] russblau: s1-user: 2w 14h 42m 0s [+0.83 s/s]; s3-rr-a: 2m 5s [+0.02 s/s]; s3-user: 2m 5s [+0.02 s/s] [22:10:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:14:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:15:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42616 MB (10% inode=99%): [22:24:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1264272 [22:39:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264467.000000 [22:39:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:39:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:40:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:41:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93224 MB (9% inode=98%): [22:41:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93224 MB (9% inode=98%): [22:41:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42597 MB (10% inode=99%): [23:10:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:24:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:35:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1266905 [23:39:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1267122.000000 [23:39:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:39:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:40:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output