[00:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:15:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:15:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:15:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:18:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5213445 [00:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78827 MB (12% inode=99%): [00:33:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90743.000000 [00:33:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:33:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:34:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:35:12] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:15:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:15:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:15:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:15:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:18:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5202402 [01:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:23:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:32:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78657 MB (12% inode=99%): [01:33:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94342.000000 [01:33:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:33:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:34:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:35:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:15:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:15:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:16:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:16:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:18:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5186769 [02:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78522 MB (12% inode=99%): [02:33:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97942.000000 [02:33:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:34:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:34:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:35:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:45:15] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [03:06:25] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [03:15:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:15:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:16:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:16:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:18:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5183950 [03:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:23:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78412 MB (12% inode=99%): [03:33:15] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101552.000000 [03:33:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:34:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:35:04] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:56:04] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:04:47] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30258 MB (7% inode=99%): [04:04:47] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 63211 MB (6% inode=97%): [04:04:55] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 63209 MB (6% inode=97%): [04:11:05] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:15:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:16:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:17:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:18:57] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5159980 [04:19:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:19:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78310 MB (12% inode=99%): [04:33:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105179.000000 [04:34:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:34:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:34:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:35:35] RAID on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:40:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [04:41:06] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [05:16:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:17:34] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:18:55] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [05:19:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78208 MB (12% inode=99%): [05:33:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [05:34:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:35:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:36:07] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:40:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [05:42:06] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [06:16:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:17:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:17:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:18:55] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [06:19:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:19:57] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:32:16] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 173463 MB (17% inode=99%): [06:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78076 MB (12% inode=99%): [06:33:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [06:34:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:35:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:40:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106678 MB (10% inode=99%): [06:40:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [06:42:07] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [06:55:06] MySQL on rosemary is OK: Uptime: 1443 Threads: 44 Questions: 349 Slow queries: 1 Opens: 56 Flush tables: 1 Open tables: 49 Queries per second avg: 0.241 [07:05:54] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3452.000000 [07:08:54] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1315.000000 [07:16:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:16:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:18:49] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:18:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:18:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:19:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:19:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77859 MB (12% inode=99%): [07:33:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115979.000000 [07:34:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:35:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:50:14] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107296 MB (11% inode=99%): [07:51:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106836 MB (10% inode=99%): [08:00:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199087 MB (20% inode=99%): [08:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:17:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:18:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68020 [08:19:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:19:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:19:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:23:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:33:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77648 MB (12% inode=99%): [08:34:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67505.000000 [08:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:35:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:35:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:17:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:18:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64255 [09:19:48] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:19:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:19:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77405 MB (12% inode=99%): [09:34:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63969.000000 [09:35:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:35:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:35:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:52:46] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:55] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:53:06] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:58:14] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:58:15] / on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:58:47] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 29937 MB (7% inode=99%): [09:58:56] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 63389 MB (6% inode=97%): [09:59:06] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 63391 MB (6% inode=97%): [09:59:06] Load avg. on z-dat-s3-a is OK: OK - load average: 0.83, 0.82, 1.10 [09:59:15] / on hyacinth is OK: DISK OK - free space: / 7456 MB (24% inode=85%): [10:02:47] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [10:02:47] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [10:03:46] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [10:03:47] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [10:05:47] MySQL on z-dat-s6-a is OK: Uptime: 432 Threads: 3 Questions: 2954 Slow queries: 1 Opens: 117 Flush tables: 1 Open tables: 109 Queries per second avg: 6.837 [10:05:47] MySQL slave on z-dat-s6-a is OK: Uptime: 432 Threads: 3 Questions: 2955 Slow queries: 1 Opens: 117 Flush tables: 1 Open tables: 109 Queries per second avg: 6.840 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 587 [10:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:17:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:18:54] nosy: ^^ some sql servers are rejecting any connection [10:19:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:19:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:19:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61195 [10:21:46] MySQL on z-dat-s3-a is OK: Uptime: 1392 Threads: 16 Questions: 2071 Slow queries: 0 Opens: 303 Flush tables: 1 Open tables: 291 Queries per second avg: 1.487 [10:21:47] MySQL slave on z-dat-s3-a is OK: Uptime: 1398 Threads: 16 Questions: 2706 Slow queries: 0 Opens: 343 Flush tables: 1 Open tables: 331 Queries per second avg: 1.935 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1587 [10:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:32:46] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2147 [10:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77144 MB (12% inode=99%): [10:34:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60003.000000 [10:35:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:35:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:35:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:40:47] MySQL slave on z-dat-s3-a is OK: Uptime: 2538 Threads: 15 Questions: 340587 Slow queries: 26 Opens: 8799 Flush tables: 1 Open tables: 8782 Queries per second avg: 134.195 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1644 [11:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:17:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:19:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:19:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:19:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57833 [11:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76865 MB (12% inode=99%): [11:34:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56832.000000 [11:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:35:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:35:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:14:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21: [12:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:19:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:19:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:19:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54341 [12:23:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76610 MB (12% inode=99%): [12:34:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53280.000000 [12:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:35:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:35:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:14:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:16:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:17:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:19:56] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47794 [13:25:43] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:25:43] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:25:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76380 MB (12% inode=99%): [13:35:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:35:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:35:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45999.000000 [13:35:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:14:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:16:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41708 [14:20:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:20:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:20:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76112 MB (12% inode=99%): [14:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:35:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:35:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39885.000000 [14:35:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:14:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:16:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34949 [15:20:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:20:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:20:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75859 MB (12% inode=99%): [15:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:35:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31674.000000 [15:35:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:36:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:14:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:17:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25667 [16:21:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:21:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:21:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:23:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:33:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75546 MB (12% inode=99%): [16:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:35:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22543.000000 [16:36:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:36:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:41:15] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:05:55] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:14:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16073 [17:21:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:21:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:21:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75250 MB (12% inode=99%): [17:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:35:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13863.000000 [17:36:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:36:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:14:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:17:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9576 [18:22:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:22:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:22:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74950 MB (12% inode=99%): [18:35:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:36:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:36:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8080.000000 [18:37:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:14:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:17:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:18:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5547 [19:22:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:22:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:23:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:33:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74649 MB (12% inode=99%): [19:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:36:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:37:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4176.000000 [19:37:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:51:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106215 MB (10% inode=99%): [20:01:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199575 MB (20% inode=99%): [20:10:34] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3591.000000 [20:10:55] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3581 [20:15:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:18:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:18:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:19:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:22:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:22:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:23:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107027 MB (11% inode=99%): [20:23:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:33:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74366 MB (12% inode=99%): [20:35:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:36:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:37:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:56:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106879 MB (10% inode=99%): [20:58:14] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107467 MB (11% inode=99%): [21:01:34] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1767.000000 [21:01:56] MySQL slave on rosemary is OK: Uptime: 52256 Threads: 10 Questions: 27572961 Slow queries: 7926 Opens: 3850 Flush tables: 1 Open tables: 631 Queries per second avg: 527.651 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1739 [21:04:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106551 MB (10% inode=99%): [21:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:16:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:18:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:18:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:19:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:22:47] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:22:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:23:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74094 MB (12% inode=99%): [21:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:36:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:37:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:46:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106913 MB (10% inode=99%): [21:58:46] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 29970 MB (7% inode=99%): [21:58:55] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 62804 MB (6% inode=97%): [21:59:06] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 62758 MB (6% inode=97%): [22:06:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107484 MB (11% inode=99%): [22:07:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106881 MB (10% inode=99%): [22:09:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107473 MB (11% inode=99%): [22:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:17:25] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:18:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:18:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:20:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:22:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:22:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:23:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:24:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:29:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106843 MB (10% inode=99%): [22:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73844 MB (12% inode=99%): [22:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:36:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:36:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:38:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:17:26] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:18:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:18:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:20:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:22:46] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:22:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:23:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:24:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:33:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73571 MB (12% inode=99%): [23:35:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:36:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:36:14] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:38:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output