[00:05:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [00:06:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:09:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:16:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:26:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 212305 MB (3% inode=23%): [00:28:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70608 MB (11% inode=99%): [00:43:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:44:49] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 18.660 second response time [00:47:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:49:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:54:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:55:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:05:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:06:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:09:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:16:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:26:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 214997 MB (4% inode=23%): [01:28:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70526 MB (11% inode=99%): [01:47:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:55:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:55:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:05:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:06:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:09:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:11:48] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 85938 MB (8% inode=99%): [02:16:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:23:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:26:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 213675 MB (4% inode=23%): [02:28:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70722 MB (11% inode=99%): [02:47:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:55:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:55:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:05:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:06:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:09:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:16:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:24:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:26:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 212110 MB (3% inode=23%): [03:28:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70256 MB (11% inode=99%): [03:47:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:55:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:55:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:07:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:09:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:17:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 211460 MB (3% inode=23%): [04:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69921 MB (11% inode=99%): [04:47:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:54:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:56:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:56:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:02:13] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [05:02:14] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [05:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:09:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:12:14] MySQL on z-dat-s3-a is OK: Uptime: 896 Threads: 4 Questions: 191 Slow queries: 0 Opens: 29 Flush tables: 1 Open tables: 22 Queries per second avg: 0.213 [05:12:14] MySQL slave on z-dat-s3-a is OK: Uptime: 896 Threads: 4 Questions: 191 Slow queries: 0 Opens: 29 Flush tables: 1 Open tables: 22 Queries per second avg: 0.213 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1056 [05:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 211871 MB (3% inode=23%): [05:28:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69752 MB (11% inode=99%): [05:47:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:56:25] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:56:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:05:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:09:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:26:35] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 211118 MB (3% inode=23%): [06:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69666 MB (11% inode=99%): [06:47:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:56:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:56:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [07:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:09:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 210931 MB (3% inode=23%): [07:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69556 MB (11% inode=99%): [07:47:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:56:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:56:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [08:07:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:09:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 210582 MB (3% inode=23%): [08:28:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69427 MB (11% inode=99%): [08:48:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:56:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:09:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:11:24] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:11:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:14:24] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:19:44] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:20:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 210111 MB (3% inode=23%): [09:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69293 MB (11% inode=99%): [09:33:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:56:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [10:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:20:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:23:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207537 MB (3% inode=23%): [10:27:55] /sql on cassia is WARNING: DISK WARNING - free space: /sql 85215 MB (7% inode=99%): [10:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69107 MB (11% inode=99%): [10:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:38:14] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 11.634 second response time [10:49:14] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2141.000000 [10:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [11:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207021 MB (3% inode=23%): [11:27:14] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1760.000000 [11:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68899 MB (11% inode=99%): [11:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 206321 MB (3% inode=23%): [12:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68698 MB (11% inode=99%): [12:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:17:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:20:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:26:36] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 205804 MB (3% inode=23%): [13:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68505 MB (11% inode=99%): [13:33:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:47:14] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1869 [13:51:14] MySQL slave on z-dat-s3-a is OK: Uptime: 32037 Threads: 22 Questions: 20460575 Slow queries: 3480 Opens: 226710 Flush tables: 1 Open tables: 16385 Queries per second avg: 638.654 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [13:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:11:54] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 85329 MB (8% inode=99%): [14:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 205227 MB (3% inode=23%): [14:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68308 MB (11% inode=99%): [14:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:23:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 204787 MB (3% inode=22%): [15:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68122 MB (11% inode=99%): [15:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:56:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:57:04] Free Memory on damiana is WARNING: WARNING - 6.6% (557060 kB) free! [16:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:06:04] Free Memory on damiana is OK: OK - 7.2% (600816 kB) free. [16:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:17:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:20:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:24:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 204485 MB (3% inode=22%): [16:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67929 MB (11% inode=99%): [16:29:04] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (394412 kB) free! [16:30:04] Free Memory on damiana is WARNING: WARNING - 5.9% (493300 kB) free! [16:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:36:03] Free Memory on damiana is OK: OK - 7.2% (603284 kB) free. [16:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:17:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:24:05] Free Memory on damiana is WARNING: WARNING - 6.8% (572472 kB) free! [17:26:04] Free Memory on damiana is OK: OK - 7.2% (601208 kB) free. [17:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 204026 MB (3% inode=22%): [17:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67715 MB (11% inode=99%): [17:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:58:04] Free Memory on damiana is WARNING: WARNING - 6.8% (566632 kB) free! [18:03:04] Free Memory on damiana is OK: OK - 7.1% (597356 kB) free. [18:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:18:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203560 MB (3% inode=22%): [18:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67481 MB (11% inode=99%): [18:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:18:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203237 MB (3% inode=22%): [19:28:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67223 MB (11% inode=99%): [19:33:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:47:04] Free Memory on damiana is WARNING: WARNING - 7.0% (584308 kB) free! [19:48:05] Free Memory on damiana is OK: OK - 7.1% (596608 kB) free. [19:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:01:14] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67092 MB (10% inode=99%): [20:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [20:07:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:18:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 202624 MB (3% inode=22%): [20:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:01:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66888 MB (10% inode=99%): [21:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [21:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:18:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 202105 MB (3% inode=22%): [21:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:01:14] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66633 MB (10% inode=99%): [22:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [22:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:17:34] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.39, 21.95, 12.62 [22:18:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:21:34] Load avg. on ortelius is WARNING: WARNING - load average: 14.27, 22.50, 15.39 [22:24:34] Load avg. on ortelius is OK: OK - load average: 3.50, 13.75, 13.16 [22:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 201634 MB (3% inode=22%): [22:27:54] /sql on cassia is WARNING: DISK WARNING - free space: /sql 87100 MB (7% inode=99%): [22:33:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:01:14] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66409 MB (10% inode=99%): [23:05:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [23:07:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:18:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:20:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:21:44] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:22:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:26:34] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 200274 MB (3% inode=22%): [23:31:04] Free Memory on damiana is WARNING: WARNING - 6.1% (507520 kB) free! [23:33:04] Free Memory on damiana is OK: OK - 7.5% (624456 kB) free. [23:33:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:56:24] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7