[00:00:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:05:06] /sql on thyme is OK: DISK OK - free space: /sql 201741 MB (21% inode=99%): [00:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:12:07] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201137 MB (20% inode=99%): [00:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117070 MB (19% inode=99%): [00:23:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:36:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:42:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13823.000000 [00:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13829 [00:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:59:40] Free Memory on damiana is WARNING: WARNING - 6.1% (507456 kB) free! [01:00:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:36] Free Memory on damiana is OK: OK - 7.1% (595632 kB) free. [01:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 117008 MB (19% inode=99%): [01:23:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:26:15] Load avg. on ortelius is WARNING: WARNING - load average: 23.65, 20.74, 14.50 [01:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:37:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:37:10] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:37:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:42:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15214.000000 [01:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15216 [01:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:52:16] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 24.88, 22.98, 20.06 [02:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:01:15] Load avg. on ortelius is WARNING: WARNING - load average: 16.60, 19.57, 19.96 [02:05:57] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116963 MB (19% inode=99%): [02:17:18] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 26.47, 21.83, 20.14 [02:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:28:16] Load avg. on ortelius is WARNING: WARNING - load average: 12.48, 18.12, 19.63 [02:34:17] Load avg. on ortelius is OK: OK - load average: 2.00, 7.95, 14.47 [02:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:37:08] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:37:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:42:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15535.000000 [02:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15541 [02:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:00:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:15:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116871 MB (19% inode=99%): [03:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:36:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:37:08] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:37:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:42:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15933.000000 [03:42:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15931 [03:43:06] /sql on thyme is OK: DISK OK - free space: /sql 201760 MB (21% inode=99%): [03:44:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:00:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:05:59] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116728 MB (19% inode=99%): [04:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:26:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:36:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:37:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:42:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16752.000000 [04:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16755 [04:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:06:55] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116612 MB (19% inode=99%): [05:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:34:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201394 MB (20% inode=99%): [05:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:36:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:37:05] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:37:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:42:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18050.000000 [05:42:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18046 [05:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:48:06] /sql on thyme is OK: DISK OK - free space: /sql 201707 MB (21% inode=99%): [05:51:26] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:55:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201350 MB (20% inode=99%): [06:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:09:05] /sql on thyme is OK: DISK OK - free space: /sql 201711 MB (21% inode=99%): [06:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116508 MB (19% inode=99%): [06:23:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:30:35] Free Memory on damiana is CRITICAL: CRITICAL - 3.8% (318536 kB) free! [06:34:36] Free Memory on damiana is WARNING: WARNING - 6.4% (534548 kB) free! [06:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:36:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201457 MB (20% inode=99%): [06:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:36] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (373832 kB) free! [06:38:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17213 [06:42:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17197.000000 [06:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:00:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116316 MB (19% inode=99%): [07:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:36:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:37:08] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:37:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15385 [07:42:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15388.000000 [07:44:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 116097 MB (19% inode=99%): [08:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:37:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:37:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:38:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14990 [08:42:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14945.000000 [08:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:30] Anyone able to kill a random query on sql-s1 for me please? ERROR 1226 (42000): User 'tb' has exceeded the 'max_user_connections' resource (current value: 15) [09:15:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115822 MB (19% inode=99%): [09:19:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 96815 MB (9% inode=99%): [09:19:36] Free Memory on damiana is WARNING: WARNING - 6.8% (567632 kB) free! [09:23:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:28:37] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419664 kB) free! [09:29:36] Free Memory on damiana is WARNING: WARNING - 6.4% (533824 kB) free! [09:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13844 [09:42:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13816.000000 [09:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:15:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115576 MB (19% inode=99%): [10:22:26] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 33132 MB (8% inode=99%): [10:23:05] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 66718 MB (6% inode=98%): [10:23:06] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 66718 MB (6% inode=98%): [10:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:36:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:38:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11656 [10:42:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11677.000000 [10:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:00:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115324 MB (19% inode=99%): [11:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:09] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:37:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:38:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:42:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10890 [11:42:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10923.000000 [11:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:58:36] Free Memory on damiana is WARNING: WARNING - 6.2% (515532 kB) free! [12:00:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:08:36] Free Memory on damiana is OK: OK - 7.1% (596764 kB) free. [12:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 115075 MB (18% inode=99%): [12:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:26:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:36:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:36:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:07] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:37:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9081 [12:42:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9022.000000 [12:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:49:35] Free Memory on damiana is WARNING: WARNING - 6.3% (530232 kB) free! [13:00:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:05:58] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114799 MB (18% inode=99%): [13:15:16] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:16:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:17:35] Free Memory on damiana is OK: OK - 7.1% (594288 kB) free. [13:23:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:26:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:29:40] select * from interwiki where interwiki.iw_prefix='commons'; [13:29:42] returns nothing? [13:35:06] Hi liangent. [13:35:49] Brooke: hi [13:36:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:36:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:37:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:38:30] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6631 [13:42:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6599.000000 [13:44:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:45:06] /sql on thyme is OK: DISK OK - free space: /sql 201949 MB (21% inode=99%): [13:59:06] @replag [13:59:07] liangent: s1-user: 1h 48m 55s [-0.00 s/s]; s2-user: 34m 38s [+0.01 s/s] [14:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114532 MB (18% inode=99%): [14:16:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:18:06] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 16.215 second response time [14:19:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198309 MB (20% inode=99%): [14:20:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:22:06] /sql on thyme is OK: DISK OK - free space: /sql 201633 MB (21% inode=99%): [14:23:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:35:08] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 18.802 second response time [14:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:37:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:41:06] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.961 second response time [14:42:06] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.627 second response time [14:42:10] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7860 [14:42:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7888.000000 [14:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:06:35] Free Memory on damiana is WARNING: WARNING - 6.9% (574348 kB) free! [15:07:08] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:08:35] Free Memory on damiana is OK: OK - 7.2% (599364 kB) free. [15:09:06] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 17.344 second response time [15:10:06] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.412 second response time [15:12:06] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.564 second response time [15:12:37] @replag [15:12:37] liangent: s1-user: 2h 19m 15s [+0.41 s/s]; s2-user: 38m 57s [+0.06 s/s]; s3-rr-a: 1m 43s [+0.00 s/s]; s3-user: 1m 43s [+0.00 s/s] [15:15:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114225 MB (18% inode=99%): [15:16:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:36:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:36:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:42:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8173 [15:42:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8190.000000 [15:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:57:59] @replag [15:58:03] Hazard-SJ: s1-user: 2h 24m 52s [+0.71 s/s]; s2-user: 40m 26s [+0.44 s/s]; s2-user-c: 5m 6s [+0.52 s/s]; s5-user-c: 5m 6s [+0.50 s/s] [16:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:07:06] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201459 MB (20% inode=99%): [16:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113984 MB (18% inode=99%): [16:16:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:24:07] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 16.578 second response time [16:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:36:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:37:06] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2146.000000 [16:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9473 [16:42:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9459.000000 [16:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:49:07] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.681 second response time [16:50:06] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 17.633 second response time [16:56:06] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 274 bytes in 20.248 second response time [17:03:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3642.000000 [17:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:15:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113719 MB (18% inode=99%): [17:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:18:05] @replag [17:18:06] liangent: s1-user: 2h 39m 30s [+0.02 s/s]; s2-user: 1h 20m 10s [+0.38 s/s]; s2-user-c: 1h 15m 46s [+0.94 s/s]; s3-rr-a: 25s [-0.00 s/s]; s3-user: 25s [-0.00 s/s]; s5-user-c: 1h 15m 46s [+0.94 s/s] [17:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:36:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:37:06] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.379 second response time [17:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:37:29] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:38:05] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 19.340 second response time [17:38:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:40:09] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.723 second response time [17:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9455 [17:42:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9473.000000 [17:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:48:36] Free Memory on damiana is WARNING: WARNING - 6.6% (554076 kB) free! [17:52:36] Free Memory on damiana is OK: OK - 7.1% (591568 kB) free. [17:57:58] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 9.772 second response time [18:03:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6595.000000 [18:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113393 MB (18% inode=99%): [18:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:36:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:37:08] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:37:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:37:11] @replag [18:37:11] liangent: s1-user: 2h 48m 44s [+0.12 s/s]; s2-user: 1h 15m 58s [-0.05 s/s]; s2-user-c: 1h 38m 40s [+0.29 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s]; s5-user-c: 1h 38m 40s [+0.29 s/s] [18:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9988 [18:42:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9971.000000 [18:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:44:36] Free Memory on damiana is WARNING: WARNING - 5.9% (497148 kB) free! [18:49:35] Free Memory on damiana is OK: OK - 7.2% (601164 kB) free. [18:52:17] Free Memory in my head is OK: OK - 86.2% (268 GB) free. [19:03:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4430.000000 [19:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:15:09] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3542.000000 [19:15:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113053 MB (18% inode=99%): [19:15:37] Free Memory on damiana is WARNING: WARNING - 6.2% (522416 kB) free! [19:16:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:24:35] Free Memory on damiana is OK: OK - 7.2% (599652 kB) free. [19:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:34:06] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [19:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:37:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:08] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:37:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:42:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10287 [19:42:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10271.000000 [19:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112738 MB (18% inode=99%): [20:16:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:10] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:37:10] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:38:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:38:36] Free Memory on damiana is WARNING: WARNING - 7.0% (586068 kB) free! [20:39:35] Free Memory on damiana is OK: OK - 7.1% (591540 kB) free. [20:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10924 [20:42:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10951.000000 [20:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:15:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112473 MB (18% inode=99%): [21:16:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:19:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 95829 MB (9% inode=99%): [21:23:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:25:35] Free Memory on damiana is WARNING: WARNING - 6.0% (499428 kB) free! [21:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:27:38] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (396372 kB) free! [21:30:36] Free Memory on damiana is WARNING: WARNING - 6.3% (524180 kB) free! [21:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:36:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:38:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:41:56] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 15.555 second response time [21:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11526 [21:43:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11564.000000 [21:44:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:05:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:11:06] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.255 second response time [22:12:07] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 16.673 second response time [22:15:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112221 MB (18% inode=99%): [22:16:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:22:26] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 33363 MB (8% inode=99%): [22:23:07] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 67284 MB (6% inode=98%): [22:23:07] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 67284 MB (6% inode=98%): [22:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:32:39] Free Memory on damiana is WARNING: WARNING - 5.9% (494460 kB) free! [22:36:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:36:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:07] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:37:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:39:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:39:35] Free Memory on damiana is OK: OK - 7.1% (593800 kB) free. [22:40:08] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 19.235 second response time [22:42:06] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.368 second response time [22:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12705 [22:42:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12696.000000 [22:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:05:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:15:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111991 MB (18% inode=99%): [23:16:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:23:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:25:05] /sql on thyme is OK: DISK OK - free space: /sql 201881 MB (21% inode=99%): [23:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:31:36] Free Memory on damiana is WARNING: WARNING - 6.0% (498944 kB) free! [23:32:05] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200665 MB (20% inode=99%): [23:34:36] Free Memory on damiana is OK: OK - 7.1% (592636 kB) free. [23:36:06] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:36:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:36:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:37:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:37:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:37:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:37:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:38:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:41:08] /sql on thyme is OK: DISK OK - free space: /sql 201716 MB (21% inode=99%): [23:42:06] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12921 [23:42:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12934.000000 [23:44:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: