[00:05:06] /sql on thyme is OK: DISK OK - free space: /sql 201650 MB (21% inode=99%): [00:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:12:08] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201297 MB (20% inode=99%): [00:14:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122270 MB (20% inode=99%): [00:22:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:36:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:36:30] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:36:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:37:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:41:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:44:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:59:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:05:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:14:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122189 MB (20% inode=99%): [01:22:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:36:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:36:28] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:36:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:37:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:41:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:44:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:59:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:05:45] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:14:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122116 MB (20% inode=99%): [02:22:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:25:06] /sql on thyme is OK: DISK OK - free space: /sql 201958 MB (21% inode=99%): [02:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:36:19] Load avg. on willow is WARNING: WARNING - load average: 17.59, 16.31, 15.16 [02:36:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:36:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:37:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:41:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:42:16] Load avg. on willow is OK: OK - load average: 13.13, 14.86, 14.91 [02:44:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:59:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:05:46] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:14:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122058 MB (20% inode=99%): [03:16:15] Load avg. on willow is WARNING: WARNING - load average: 16.83, 16.31, 15.04 [03:19:18] Load avg. on willow is OK: OK - load average: 13.54, 14.82, 14.66 [03:22:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:36:16] Load avg. on willow is WARNING: WARNING - load average: 16.21, 16.15, 15.53 [03:36:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:36:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:37:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:37:09] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:37:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:41:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:44:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:50:16] Load avg. on willow is OK: OK - load average: 12.10, 13.98, 14.89 [03:59:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:01:13] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3354 MB (19% inode=99%): [04:04:06] / on willow is WARNING: DISK WARNING - free space: / 21979 MB (20% inode=99%): [04:04:06] /tmp on willow is WARNING: DISK WARNING - free space: / 21979 MB (20% inode=99%): [04:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:14:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121996 MB (20% inode=99%): [04:22:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:36:06] /tmp on ortelius is OK: DISK OK - free space: /tmp 3768 MB (21% inode=99%): [04:36:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:36:28] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:37:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:10] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:37:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:37:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:41:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:44:09] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3419 MB (19% inode=99%): [04:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:47:08] /tmp on ortelius is OK: DISK OK - free space: /tmp 3780 MB (21% inode=99%): [04:59:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:04:06] / on willow is WARNING: DISK WARNING - free space: / 21921 MB (20% inode=99%): [05:04:06] /tmp on willow is WARNING: DISK WARNING - free space: / 21921 MB (20% inode=99%): [05:05:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121888 MB (20% inode=99%): [05:21:50] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:22:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:22:35] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:23:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:23:36] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:26:06] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3552 MB (20% inode=99%): [05:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:36:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:36:28] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:36:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:37:06] /tmp on ortelius is OK: DISK OK - free space: /tmp 3719 MB (21% inode=99%): [05:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:37:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:47:06] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3622 MB (20% inode=99%): [05:49:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201460 MB (20% inode=99%): [05:57:06] /sql on thyme is OK: DISK OK - free space: /sql 201783 MB (21% inode=99%): [06:04:09] / on willow is WARNING: DISK WARNING - free space: / 21866 MB (20% inode=99%): [06:04:09] /tmp on willow is WARNING: DISK WARNING - free space: / 21866 MB (20% inode=99%): [06:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:15:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201412 MB (20% inode=99%): [06:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121715 MB (20% inode=99%): [06:22:06] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3549 MB (20% inode=99%): [06:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:24:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:37:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:37:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:38:08] /sql on thyme is OK: DISK OK - free space: /sql 201527 MB (21% inode=99%): [06:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:47:07] / on willow is OK: DISK OK - free space: / 31339 MB (29% inode=99%): [06:47:07] /tmp on willow is OK: DISK OK - free space: / 31339 MB (29% inode=99%): [06:48:08] /tmp on ortelius is OK: DISK OK - free space: /tmp 12204 MB (70% inode=99%): [06:54:38] Free Memory on damiana is WARNING: WARNING - 7.0% (583584 kB) free! [06:55:36] Free Memory on damiana is OK: OK - 7.2% (603120 kB) free. [07:04:38] Free Memory on damiana is WARNING: WARNING - 6.4% (533800 kB) free! [07:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121527 MB (20% inode=99%): [07:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:24:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:30:07] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201436 MB (20% inode=99%): [07:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:36:29] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:36:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:44:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:05:06] @replag [08:05:07] Merlissimo: s1-user: 30s [-0.00 s/s]; s3-rr-a: 10s [-0.00 s/s]; s3-user: 10s [-0.00 s/s] [08:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121260 MB (19% inode=99%): [08:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:24:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:36:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:37:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:06:00] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:06:35] Free Memory on damiana is WARNING: WARNING - 6.9% (575352 kB) free! [09:08:37] Free Memory on damiana is OK: OK - 7.3% (609684 kB) free. [09:15:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120961 MB (19% inode=99%): [09:19:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 96879 MB (9% inode=99%): [09:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:24:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:36:26] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:38:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120719 MB (19% inode=99%): [10:22:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:22:26] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 32693 MB (8% inode=99%): [10:23:07] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 65084 MB (6% inode=97%): [10:23:07] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 65084 MB (6% inode=97%): [10:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:24:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:36:27] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:37:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:09] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:37:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:39] Free Memory on damiana is WARNING: WARNING - 6.3% (529204 kB) free! [11:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120457 MB (19% inode=99%): [11:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:24:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:28:36] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (403352 kB) free! [11:29:36] Free Memory on damiana is WARNING: WARNING - 6.1% (515084 kB) free! [11:34:06] /sql on thyme is OK: DISK OK - free space: /sql 202151 MB (21% inode=99%): [11:36:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:05:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:12:39] hello all [12:12:40] @replag [12:12:41] DaBPunkt: s3-rr-a: 12m 38s [+0.05 s/s]; s3-user: 12m 38s [+0.05 s/s]; s6-rr-a: 43s [+0.00 s/s]; s6-user: 43s [+0.00 s/s]; s7-rr-a: 17m 5s [+0.01 s/s]; s7-user: 17m 5s [+0.01 s/s] [12:15:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120162 MB (19% inode=99%): [12:16:45] hi DaBPunkt. [12:22:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:24:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:36:06] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2016 [12:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:37:29] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:38:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:42:07] MySQL slave on z-dat-s7-a is OK: Uptime: 1382524 Threads: 8 Questions: 392663353 Slow queries: 25597 Opens: 3225357 Flush tables: 1 Open tables: 8248 Queries per second avg: 284.19 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1779 [12:44:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:58:15] Sun Grid Engine execd on wolfsbane is UNKNOWN: NRPE: Unable to read output [12:59:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119878 MB (19% inode=99%): [13:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:24:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:36:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:36:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:00:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119626 MB (19% inode=99%): [14:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:24:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:36:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:36:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:49:35] Free Memory on damiana is WARNING: WARNING - 6.7% (559548 kB) free! [14:54:35] Free Memory on damiana is OK: OK - 7.2% (606244 kB) free. [14:57:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201440 MB (20% inode=99%): [15:00:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:15:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119389 MB (19% inode=99%): [15:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:24:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:36:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:45:06] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1973 [15:45:45] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1986.000000 [16:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:04:36] Free Memory on damiana is WARNING: WARNING - 6.8% (571216 kB) free! [16:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:08:35] Free Memory on damiana is OK: OK - 7.3% (607948 kB) free. [16:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119085 MB (19% inode=99%): [16:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:25:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:36:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:37:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:45:06] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3294 [16:45:46] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3330.000000 [17:00:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:05:09] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3601 [17:05:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [17:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118829 MB (19% inode=99%): [17:17:45] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3597.000000 [17:18:06] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3589 [17:20:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3611.000000 [17:21:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3602 [17:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:25:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:29:36] Free Memory on damiana is WARNING: WARNING - 6.5% (548564 kB) free! [17:32:36] Free Memory on damiana is OK: OK - 7.1% (593460 kB) free. [17:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:37:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:41:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4262.000000 [17:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4281 [17:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:49:35] Free Memory on damiana is WARNING: WARNING - 6.4% (538688 kB) free! [17:56:37] Free Memory on damiana is OK: OK - 7.1% (593716 kB) free. [18:00:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:14:26] Load avg. on willow is WARNING: WARNING - load average: 13.62, 16.26, 13.94 [18:15:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118562 MB (19% inode=99%): [18:17:25] Load avg. on willow is OK: OK - load average: 11.12, 14.62, 13.75 [18:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:25:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:41:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5310.000000 [18:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5320 [18:44:06] /sql on thyme is OK: DISK OK - free space: /sql 202066 MB (21% inode=99%): [18:44:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:00:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:05:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:15:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118307 MB (19% inode=99%): [19:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:26:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:31:11] @replag [19:31:12] jimmyxu: s1-user: 1h 53m 21s [+0.16 s/s]; s3-rr-a: 41s [-0.03 s/s]; s3-user: 41s [-0.03 s/s] [19:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:41:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6816.000000 [19:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6830 [19:44:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:00:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:12:39] Free Memory on damiana is WARNING: WARNING - 6.9% (575916 kB) free! [20:14:37] Free Memory on damiana is OK: OK - 7.1% (593000 kB) free. [20:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117940 MB (19% inode=99%): [20:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:26:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:36:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:41:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7300.000000 [20:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7309 [20:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:51:35] Free Memory on damiana is WARNING: WARNING - 7.0% (585128 kB) free! [20:55:36] Free Memory on damiana is OK: OK - 7.2% (601524 kB) free. [21:00:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117611 MB (19% inode=99%): [21:17:08] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201457 MB (20% inode=99%): [21:19:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 96751 MB (9% inode=99%): [21:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:24:36] Free Memory on damiana is WARNING: WARNING - 6.7% (562640 kB) free! [21:26:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:26:35] Free Memory on damiana is OK: OK - 7.2% (599660 kB) free. [21:36:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:36:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:37:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:07] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:37:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:41:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9136.000000 [21:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9149 [21:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:00:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:06:00] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117372 MB (19% inode=99%): [22:22:27] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 32950 MB (8% inode=99%): [22:23:05] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 67088 MB (6% inode=98%): [22:23:11] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 67088 MB (6% inode=98%): [22:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:36:41] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:10] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:37:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:42:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10584.000000 [22:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10587 [22:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:00:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:15:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117193 MB (19% inode=99%): [23:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:26:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:36:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:36:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:42:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12844.000000 [23:42:08] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12841 [23:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: