[00:06:30] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2489 MB (18% inode=99%): [00:09:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:11:34] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 63127 MB (6% inode=97%): [00:12:37] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 63129 MB (6% inode=97%): [00:19:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:23:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:24:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:25:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:26:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:27:50] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:28:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:29:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:31:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:31:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:33:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:37:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64695 MB (10% inode=99%): [00:43:28] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:51:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:06:29] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2562 MB (19% inode=99%): [01:10:01] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:14:31] /tmp on ortelius is OK: DISK OK - free space: /tmp 2855 MB (21% inode=99%): [01:19:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:21:29] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2482 MB (18% inode=99%): [01:23:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:25:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:25:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:26:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:27:28] /tmp on ortelius is OK: DISK OK - free space: /tmp 3032 MB (22% inode=99%): [01:27:51] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:28:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:29:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:31:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:31:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:33:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:37:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64595 MB (10% inode=99%): [01:43:29] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:51:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:05:32] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2286 MB (17% inode=99%): [02:10:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:19:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:23:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:25:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:25:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:26:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:27:50] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:28:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:29:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:32:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:32:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:33:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:37:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64463 MB (10% inode=99%): [02:41:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:43:28] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:51:10] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:01:29] /tmp on ortelius is OK: DISK OK - free space: /tmp 2898 MB (21% inode=99%): [03:11:02] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:19:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:21:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:23:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:25:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:26:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:27:52] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:28:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:29:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:32:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:32:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:35:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:37:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64310 MB (10% inode=99%): [03:43:29] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:02:31] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2354 MB (18% inode=99%): [04:11:00] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:23] aliasd on yarrow is CRITICAL: Connection refused [04:20:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:21:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:23:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:25:26] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:26:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:26:23] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:28:03] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:28:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:29:31] /tmp on ortelius is OK: DISK OK - free space: /tmp 2830 MB (21% inode=99%): [04:30:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:33:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:33:21] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:35:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:38:00] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64158 MB (10% inode=99%): [04:43:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:47:43] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2760 MB (20% inode=99%): [04:53:42] /tmp on ortelius is OK: DISK OK - free space: /tmp 2858 MB (21% inode=99%): [05:11:26] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:14:20] aliasd on yarrow is CRITICAL: Connection refused [05:20:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:23:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:25:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:26:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:27:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:28:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:29:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:30:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:33:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:34:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:35:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:38:00] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64009 MB (10% inode=99%): [05:43:55] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:49:42] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2583 MB (19% inode=99%): [06:11:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:14:21] aliasd on yarrow is CRITICAL: Connection refused [06:21:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:23:46] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:25:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:26:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:27:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:28:00] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:29:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:30:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:33:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:34:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:35:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:38:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63867 MB (10% inode=99%): [06:44:51] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:50:43] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2479 MB (18% inode=99%): [06:51:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:11:21] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:15:23] aliasd on yarrow is CRITICAL: Connection refused [07:21:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:24:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:25:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:27:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:28:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:28:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:29:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:31:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:33:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:35:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:36:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:38:00] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63652 MB (10% inode=99%): [07:45:51] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:51:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:51:43] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2335 MB (18% inode=99%): [08:02:00] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198552 MB (20% inode=99%): [08:11:21] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:15:21] aliasd on yarrow is CRITICAL: Connection refused [08:21:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:24:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:26:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:27:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:28:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:29:02] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:29:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:31:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:33:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:35:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:36:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:38:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63406 MB (10% inode=99%): [08:44:41] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1258 MB (10% inode=99%): [08:45:00] /sql on ptolemy is OK: DISK OK - free space: /sql 177005 MB (29% inode=99%): [08:45:42] /tmp on ortelius is OK: DISK OK - free space: /tmp 8338 MB (69% inode=99%): [08:45:51] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:51:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:11:20] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:22] aliasd on yarrow is CRITICAL: Connection refused [09:16:20] aliasd on yarrow is OK: TCP OK - 0.012 second response time on port 984 [500 Not found.] [09:21:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:24:42] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:26:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:27:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:28:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:29:02] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:29:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:31:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:34:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:36:21] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:36:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:46:53] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:03:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105420 MB (10% inode=99%): [10:11:21] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:21:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:22:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:25:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:26:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:27:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:29:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:29:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:30:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:31:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:34:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:36:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:37:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:47:51] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:12:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:21:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:22:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:25:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:26:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:28:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:29:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:29:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:30:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:32:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:34:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:36:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:37:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:44:51] hello all [11:46:21] I will stop replication on s3 for some minutes [11:47:51] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:48:10] and done [11:51:16] stopping s7 for a short moment [11:58:30] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [11:59:07] @replag [11:59:08] DaBPunkt: s3-rr-a: 39s [-0.03 s/s]; s3-user: 39s [-0.03 s/s]; s7-rr-a: 7m 46s [+0.00 s/s]; s7-user: 7m 46s [+0.00 s/s] [11:59:31] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 29182 MB (7% inode=99%): [12:07:31] MySQL slave on z-dat-s7-a is OK: Uptime: 781120 Threads: 5 Questions: 257285217 Slow queries: 13832 Opens: 2224720 Flush tables: 1 Open tables: 5349 Queries per second avg: 329.379 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 710 [12:11:50] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 63051 MB (6% inode=97%): [12:12:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:12:50] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 63040 MB (6% inode=97%): [12:22:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:25:44] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:26:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:28:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:29:05] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:29:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:30:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:32:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:35:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:36:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:37:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:47:50] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:51:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:57:40] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [12:58:33] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1991.000000 [13:01:30] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2213 [13:01:30] aliasd on yarrow is CRITICAL: Connection refused [13:06:55] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [13:09:31] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [13:09:50] restart msql on thyme because it swaps [13:12:20] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:14:32] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2962 [13:16:31] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107043 MB (11% inode=99%): [13:18:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105610 MB (10% inode=99%): [13:20:30] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108724 MB (11% inode=99%): [13:23:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:24:58] @reolag [13:24:58] @replag [13:24:58] Betacommand: s1-rr-a: 31m 2s [-0.01 s/s]; s3-rr-a: 17s [-0.00 s/s]; s3-user: 17s [-0.00 s/s] [13:26:34] MySQL slave on thyme is OK: Uptime: 866 Threads: 7 Questions: 169701 Slow queries: 77 Opens: 643 Flush tables: 1 Open tables: 121 Queries per second avg: 195.959 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1681 [13:26:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:26:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:29:30] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:29:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:30:00] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:30:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:32:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:35:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:36:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:37:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:47:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106480 MB (10% inode=99%): [13:47:50] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:51:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:01:30] aliasd on yarrow is CRITICAL: Connection refused [14:12:20] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:14:09] @replag [14:14:09] DaBPunkt: s2-user: 6m 15s [+0.00 s/s]; s2-user-c: 2m 12s [+0.00 s/s]; s5-user-c: 2m 12s [+0.00 s/s] [14:14:20] @replag [14:14:20] DaBPunkt: s2-user: 6m 13s [-0.23 s/s]; s2-user-c: 1m 58s [-1.64 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s]; s5-user-c: 1m 58s [-1.64 s/s] [14:15:59] DaBPunkt: is s1 corruption a known issue? [14:16:18] Betacommand: yes. That's the reason we requested a new dump [14:16:30] I'm compiling the software at the moment [14:17:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108034 MB (11% inode=99%): [14:19:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106543 MB (10% inode=99%): [14:21:02] Environment IPMI on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:22:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:23:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:26:40] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:27:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:28:31] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107217 MB (11% inode=99%): [14:29:31] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:30:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:30:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:30:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:33:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:35:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:36:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106895 MB (10% inode=99%): [14:37:20] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:48:01] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:51:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:01:11] DaB. * [Toolserver-announce] Welcome: wikidata [15:01:31] aliasd on yarrow is CRITICAL: Connection refused [15:03:29] @replag [15:03:30] DaBPunkt: s1-user-wd: 46m 57s [-]; s2-user: 19s [-]; s3-rr-a: 54s [-]; s3-user: 54s [-] [15:03:41] @replag [15:03:41] DaBPunkt: s1-user-wd: 47m 9s [+1.08 s/s]; s3-rr-a: 10s [-3.95 s/s]; s3-user: 10s [-3.95 s/s] [15:06:38] @replag [15:06:38] DaBPunkt: s1-rr-a-wd: 30m 43s [-]; s1-user-wd: 50m 6s [-]; s2-user-wd: 25m 39s [-]; s3-rr-a: 10s [-]; s3-user: 10s [-]; s4-user-wd: 50m 6s [-]; s5-user-wd: 25m 39s [-]; s6-user-wd: error [15:06:39] DaBPunkt: s7-user-wd: error [15:07:25] @replag [15:07:25] DaBPunkt: s1-rr-a-wd: 31m 30s [+1.01 s/s]; s1-user-wd: 50m 53s [+1.01 s/s]; s2-user-wd: 26m 26s [+1.01 s/s]; s3-rr-a: 42s [+0.69 s/s]; s3-user: 42s [+0.69 s/s]; s4-user-wd: 50m 53s [+1.01 s/s]; s5-user-wd: 26m 26s [+1.01 s/s]; s6-user-wd: error [15:07:29] DaBPunkt: s7-user-wd: error [15:22:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:23:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:24:06] @replag [15:24:06] DaBPunkt: s1-rr-a-wd: 48m 10s [-]; s1-user-wd: 5m 34s [-]; s2-user-wd: 43m 6s [-]; s4-user-wd: 1h 7m 33s [-]; s5-user-wd: 43m 6s [-] [15:24:27] @replag [15:24:27] DaBPunkt: s1-rr-a-wd: 48m 32s [+1.02 s/s]; s1-user-wd: 5m 12s [-1.01 s/s]; s2-user-wd: 43m 28s [+1.01 s/s]; s3-rr-a: 29s [-]; s3-user: 29s [-]; s4-user-wd: 1h 7m 55s [+1.00 s/s]; s5-user-wd: 43m 28s [+1.00 s/s] [15:25:03] What is -wd? [15:25:08] Brooke: wikidata [15:25:12] Ahh. [15:26:08] Magnus Manske * Re: [Toolserver-l] Welcome: wikidata [15:26:44] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:27:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:27:43] @replag [15:27:43] DaBPunkt: s1-rr-a-wd: 7m 2s [-12.69 s/s]; s1-user-wd: 4m 37s [-0.18 s/s]; s2-user-wd: 46m 44s [+1.00 s/s]; s3-rr-a: 14s [-0.08 s/s]; s3-user: 14s [-0.08 s/s]; s4-user-wd: 1h 11m 11s [+1.00 s/s]; s5-user-wd: 46m 44s [+1.00 s/s] [15:29:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:30:03] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:30:05] @replag [15:30:06] DaBPunkt: s1-user-wd: 2m 39s [-0.83 s/s]; s2-user-wd: 49m 6s [+1.00 s/s]; s3-rr-a: 21s [+0.05 s/s]; s3-user: 21s [+0.05 s/s]; s4-user-wd: 1h 13m 33s [+1.00 s/s]; s5-user-wd: 49m 7s [+1.00 s/s] [15:30:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:30:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:33:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:33:25] @replag [15:33:25] DaBPunkt: s1-user-wd: 28s [-0.66 s/s]; s2-user-wd: 52m 24s [+1.00 s/s]; s3-rr-a: 60s [+0.20 s/s]; s3-user: 60s [+0.20 s/s]; s4-user-wd: 1h 16m 51s [+1.00 s/s]; s5-user-wd: 52m 24s [+1.00 s/s] [15:34:59] @replag all [15:35:04] Brooke: s1-rr-a: 0s [-]; s1-rr-a-wd: 13s [-0.94 s/s]; s1-user: 0s [-]; s1-user-c: 1s [-]; s1-user-wd: 31s [+0.03 s/s]; s2-user: 9s [-]; s2-user-c: 1s [-]; s2-user-wd: 54m 0s [+0.99 s/s] [15:35:04] Brooke: s3-rr-a: 9s [-0.53 s/s]; s3-user: 9s [-0.53 s/s]; s4-rr-a: 1s [-]; s4-user: 1s [-]; s4-user-wd: 1h 18m 27s [+0.99 s/s]; s5-rr-a: 4s [-]; s5-user: 4s [-]; s5-user-c: 2s [-] [15:35:04] Brooke: s5-user-wd: 54m 1s [+1.00 s/s]; s6-rr-a: 1s [-]; s6-user: 1s [-]; s7-rr-a: 1s [-]; s7-user: 1s [-] [15:35:10] How things have grown. (o: [15:35:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:37:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:38:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:41:09] @replag [15:41:09] DaBPunkt: s1-user-wd: 27s [-0.01 s/s]; s2-user-wd: 1h 10s [+1.00 s/s]; s4-user-wd: 15m 59s [-10.15 s/s]; s5-user-wd: 1h 10s [+1.00 s/s] [15:43:45] @replag [15:43:48] DaBPunkt: s1-rr-a-wd: 10s [-0.01 s/s]; s1-user-wd: 25s [-0.01 s/s]; s2-user: 1m 5s [+0.11 s/s]; s2-user-c: 52s [+0.10 s/s]; s2-user-wd: 26m 9s [-13.02 s/s]; s3-rr-a: 50s [+0.08 s/s]; s3-user: 50s [+0.08 s/s]; s4-user-wd: 11s [-6.01 s/s] [15:43:49] DaBPunkt: s5-rr-a: 48s [+0.08 s/s]; s5-user: 50s [+0.09 s/s]; s5-user-c: 51s [+0.09 s/s]; s5-user-wd: 26m 12s [-12.83 s/s] [15:48:00] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:50:23] @replag [15:50:24] DaBPunkt: s2-user-wd: 30s [-3.87 s/s]; s4-user-wd: 17s [+0.02 s/s]; s5-user-wd: 30s [-3.90 s/s]; s7-rr-a: 10s [+0.01 s/s]; s7-user: 10s [+0.01 s/s] [15:57:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105183 MB (10% inode=99%): [16:01:31] aliasd on yarrow is CRITICAL: Connection refused [16:21:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:22:08] DaB. * Re: [Toolserver-l] Welcome: wikidata [16:22:42] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:26:40] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:27:53] @replag [16:27:53] DaBPunkt: s1-rr-a-wd: 37s [+0.01 s/s]; s1-user-wd: 22s [-0.00 s/s]; s3-rr-a: 24s [-0.01 s/s]; s3-user: 24s [-0.01 s/s]; s4-user-wd: 37s [+0.01 s/s] [16:28:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:29:31] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:30:00] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:30:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:30:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:33:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:35:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:38:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:48:00] @replag [16:48:01] legoktm: s1-rr-a-wd: 32s [-0.00 s/s]; s1-user-wd: 26s [+0.00 s/s]; s2-user-wd: 24s [-0.00 s/s]; s3-rr-a: 13s [-0.01 s/s]; s3-user: 13s [-0.01 s/s]; s5-user-wd: 24s [-0.00 s/s] [16:48:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:01:35] aliasd on yarrow is CRITICAL: Connection refused [17:17:31] Load avg. on ortelius is WARNING: WARNING - load average: 10.40, 23.93, 18.02 [17:21:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:21:33] Load avg. on ortelius is OK: OK - load average: 3.27, 12.82, 14.71 [17:22:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:26:40] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:28:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:29:31] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:30:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:30:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:31:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:33:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:35:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:39:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:49:01] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:01:32] aliasd on yarrow is CRITICAL: Connection refused [18:15:40] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:16:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:20:11] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:20:40] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:20:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:21:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:22:32] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:25:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:26:40] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:28:31] MySQL on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [18:28:31] MySQL slave on cassia is CRITICAL: Cant connect to MySQL server on cassia (146) [18:28:31] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [18:28:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:29:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146) [18:29:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:30:09] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:31:06] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:31:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:38:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:39:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:40:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:41] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:42:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:42:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:45:35] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107037 MB (11% inode=99%): [18:46:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105265 MB (10% inode=99%): [18:49:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:49:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:54:20] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1919.000000 [18:54:36] MySQL on cassia is OK: Uptime: 1917 Threads: 46 Questions: 411 Slow queries: 11 Opens: 117 Flush tables: 1 Open tables: 94 Queries per second avg: 0.214 [18:54:36] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1933 [18:54:36] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1939.000000 [18:57:33] MySQL slave on cassia is OK: Uptime: 2099 Threads: 19 Questions: 259872 Slow queries: 69 Opens: 869 Flush tables: 1 Open tables: 773 Queries per second avg: 123.807 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1619 [18:57:33] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1611.000000 [19:00:13] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1625.000000 [19:02:01] Environment IPMI on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:02:31] aliasd on yarrow is CRITICAL: Connection refused [19:02:40] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:03:31] hmm. array jobs on linux machines is still broken. [19:03:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:04:32] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:08:21] Environment IPMI on wolfsbane is UNKNOWN: NRPE: Unable to read output [19:08:40] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:13:31] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 38.39, 23.09, 13.07 [19:15:31] /tmp on wolfsbane is CRITICAL: Connection refused by host [19:15:31] Load avg. on ortelius is WARNING: WARNING - load average: 15.67, 21.23, 13.74 [19:16:40] Load avg. on wolfsbane is CRITICAL: Connection refused by host [19:17:33] / on wolfsbane is CRITICAL: Connection refused by host [19:17:33] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 49.41, 27.66, 16.80 [19:26:42] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:28:00] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [19:28:40] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [19:28:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:30:31] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:31:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:31:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:34:34] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108302 MB (11% inode=99%): [19:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:36:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106194 MB (10% inode=99%): [19:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:39:33] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107148 MB (11% inode=99%): [19:39:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:42:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:49:00] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:51:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:56:36] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 26.55, 29.74, 20.37 [19:58:31] Load avg. on ortelius is WARNING: WARNING - load average: 9.30, 22.16, 18.66 [20:02:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198474 MB (20% inode=99%): [20:02:31] aliasd on yarrow is CRITICAL: Connection refused [20:02:31] Load avg. on ortelius is OK: OK - load average: 1.54, 11.14, 14.84 [20:15:31] /tmp on wolfsbane is CRITICAL: Connection refused by host [20:16:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105854 MB (10% inode=99%): [20:16:40] Load avg. on wolfsbane is CRITICAL: Connection refused by host [20:17:33] / on wolfsbane is CRITICAL: Connection refused by host [20:22:34] /tmp on wolfsbane is OK: DISK OK - free space: / 8505 MB (28% inode=93%): [20:22:34] / on wolfsbane is OK: DISK OK - free space: / 8505 MB (28% inode=93%): [20:22:41] Load avg. on wolfsbane is OK: OK - load average: 1.03, 0.80, 0.73 [20:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:27:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:28:32] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:28:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:30:31] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:31:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:31:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:33:32] aliasd on yarrow is OK: TCP OK - 0.008 second response time on port 984 [500 Not found.] [20:35:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:39:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:42:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:43:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:49:01] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:51:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:23:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:27:40] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:28:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:29:31] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:31:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:31:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:31:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:36:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:38:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:39:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:43:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:49:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:51:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:23:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:27:41] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:28:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:29:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:31:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:32:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:32:32] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:36:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:34] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:40:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:43:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:43:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:49:01] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:21:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:27:40] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:29:32] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:29:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:31:01] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:32:31] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:32:31] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:36:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:39:34] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:40:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:43:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:43:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:49:01] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:54:30] gn8 toolserver [23:54:31] aliasd on yarrow is CRITICAL: Connection refused [23:59:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30046 MB (7% inode=99%):