[00:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:07:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:07:13] Free Memory on damiana is WARNING: WARNING - 5.6% (465100 kB) free! [00:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:10:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:14:13] Free Memory on damiana is OK: OK - 7.1% (591532 kB) free. [00:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 262364 MB (4% inode=55%): [00:17:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.64, 21.89, 21.61 [00:18:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 887913.000000 [00:30:06] nacht ts [00:34:13] Free Memory on damiana is WARNING: WARNING - 6.2% (517788 kB) free! [00:36:53] /sql on cassia is OK: DISK OK - free space: /sql 130619 MB (11% inode=99%): [00:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:37:13] Free Memory on damiana is OK: OK - 7.1% (596228 kB) free. [00:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124307 MB (20% inode=99%): [00:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:53:14] Load avg. on willow is WARNING: WARNING - load average: 18.61, 19.39, 19.94 [00:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:01:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.44, 23.02, 21.09 [01:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 265044 MB (4% inode=55%): [01:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 891513.000000 [01:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122603 MB (20% inode=99%): [01:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:01:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.11, 22.07, 20.93 [02:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 264582 MB (4% inode=55%): [02:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 895113.000000 [02:21:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 77778 MB (7% inode=99%): [02:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124042 MB (20% inode=99%): [02:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:01:14] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.84, 23.89, 22.29 [03:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:07:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:08:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:10:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:11:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 130032 MB (10% inode=99%): [03:15:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 264144 MB (4% inode=55%): [03:18:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 898712.000000 [03:25:53] /sql on cassia is OK: DISK OK - free space: /sql 130138 MB (11% inode=99%): [03:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:38:14] Free Memory on damiana is WARNING: WARNING - 5.3% (444468 kB) free! [03:39:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123956 MB (20% inode=99%): [03:44:13] Free Memory on damiana is OK: OK - 8.3% (697976 kB) free. [03:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:52:53] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129452 MB (10% inode=99%): [03:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:54:53] /sql on cassia is OK: DISK OK - free space: /sql 131121 MB (11% inode=99%): [04:00:14] Free Memory on damiana is WARNING: WARNING - 5.2% (434284 kB) free! [04:01:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.79, 24.55, 23.64 [04:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:07:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:11:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128804 MB (10% inode=99%): [04:11:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:16:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263209 MB (4% inode=55%): [04:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 902319.000000 [04:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:37:22] Load avg. on nightshade is WARNING: WARNING - load average: 15.72, 15.68, 14.41 [04:40:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123902 MB (20% inode=99%): [04:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:01:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.11, 25.49, 24.01 [05:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:07:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:08:01] /sql on cassia is OK: DISK OK - free space: /sql 132007 MB (11% inode=99%): [05:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:11:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:15:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126231 MB (10% inode=99%): [05:16:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 264187 MB (4% inode=55%): [05:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 905919.000000 [05:23:01] /sql on cassia is OK: DISK OK - free space: /sql 132255 MB (11% inode=99%): [05:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:37:22] Load avg. on nightshade is WARNING: WARNING - load average: 14.74, 15.84, 16.66 [05:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123832 MB (20% inode=99%): [05:47:22] Load avg. on nightshade is OK: OK - load average: 11.51, 13.06, 15.00 [05:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:01:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.10, 25.96, 23.63 [06:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:11:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:16:21] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263932 MB (4% inode=55%): [06:18:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 909519.000000 [06:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:40:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129947 MB (10% inode=99%): [06:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123717 MB (20% inode=99%): [06:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:01:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.83, 23.41, 22.25 [07:02:21] Free Memory on damiana is WARNING: WARNING - 6.2% (519516 kB) free! [07:03:00] /sql on cassia is OK: DISK OK - free space: /sql 130421 MB (11% inode=99%): [07:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:10:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129200 MB (10% inode=99%): [07:11:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:12:22] Load avg. on nightshade is WARNING: WARNING - load average: 29.31, 23.03, 17.81 [07:15:21] Free Memory on damiana is OK: OK - 7.2% (604268 kB) free. [07:16:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263694 MB (4% inode=55%): [07:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 913118.000000 [07:27:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 23.33, 21.54, 20.18 [07:37:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123602 MB (20% inode=99%): [07:43:21] Load avg. on nightshade is WARNING: WARNING - load average: 15.24, 18.25, 19.82 [07:45:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 22.91, 19.72, 20.12 [07:47:21] Load avg. on nightshade is WARNING: WARNING - load average: 17.78, 19.32, 19.96 [07:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:01:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.08, 25.51, 22.67 [08:01:22] Free Memory on damiana is WARNING: WARNING - 5.1% (426720 kB) free! [08:02:22] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (414988 kB) free! [08:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:04:01] /sql on cassia is OK: DISK OK - free space: /sql 130132 MB (11% inode=99%): [08:05:21] Free Memory on damiana is WARNING: WARNING - 5.4% (450788 kB) free! [08:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:09:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:11:00] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129302 MB (10% inode=99%): [08:11:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:13:22] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 27.07, 23.33, 20.31 [08:16:22] Free Memory on damiana is OK: OK - 7.2% (601828 kB) free. [08:16:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263426 MB (4% inode=55%): [08:18:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 916719.000000 [08:37:01] /sql on cassia is OK: DISK OK - free space: /sql 131958 MB (11% inode=99%): [08:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123438 MB (20% inode=99%): [08:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:51:21] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (408760 kB) free! [08:53:21] Free Memory on damiana is WARNING: WARNING - 5.1% (427940 kB) free! [09:01:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.30, 22.99, 21.78 [09:03:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128868 MB (10% inode=99%): [09:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:06:22] Load avg. on nightshade is WARNING: WARNING - load average: 13.65, 18.18, 19.73 [09:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:08:22] Free Memory on damiana is OK: OK - 7.5% (627520 kB) free. [09:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:11:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:17:22] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 262806 MB (4% inode=55%): [09:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 920319.000000 [09:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:33:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 23.03, 21.47, 20.30 [09:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123209 MB (20% inode=99%): [09:48:21] Free Memory on damiana is WARNING: WARNING - 5.9% (497440 kB) free! [09:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:01:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.05, 22.75, 21.61 [10:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:06:21] Free Memory on damiana is OK: OK - 7.1% (593720 kB) free. [10:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:09:21] Load avg. on nightshade is WARNING: WARNING - load average: 17.14, 18.82, 19.85 [10:11:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:17:21] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 262155 MB (4% inode=55%): [10:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 923918.000000 [10:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:37:22] Free Memory on damiana is WARNING: WARNING - 5.6% (470888 kB) free! [10:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122976 MB (20% inode=99%): [10:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:59:21] Free Memory on damiana is OK: OK - 7.2% (602324 kB) free. [11:01:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.50, 24.64, 22.64 [11:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:07:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:09:22] Load avg. on nightshade is WARNING: WARNING - load average: 19.21, 16.88, 17.36 [11:11:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:17:21] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 260165 MB (4% inode=55%): [11:18:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 927518.000000 [11:24:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122722 MB (20% inode=99%): [11:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:59:30] Free Memory on damiana is WARNING: WARNING - 6.8% (566928 kB) free! [12:01:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.55, 24.84, 22.66 [12:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:08:31] Free Memory on damiana is OK: OK - 7.3% (607620 kB) free. [12:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:10:03] Chris Grant * Re: [Toolserver-l] Mail forwarding not working [12:10:22] Load avg. on nightshade is WARNING: WARNING - load average: 17.77, 17.57, 17.70 [12:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 259497 MB (4% inode=55%): [12:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931119.000000 [12:30:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (378000 kB) free! [12:34:31] Free Memory on damiana is WARNING: WARNING - 5.3% (441652 kB) free! [12:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122452 MB (20% inode=99%): [12:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:58:31] Free Memory on damiana is OK: OK - 7.2% (603820 kB) free. [13:01:32] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.41, 23.43, 22.95 [13:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:05:31] Free Memory on damiana is WARNING: WARNING - 5.3% (443168 kB) free! [13:06:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (392436 kB) free! [13:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:10:21] Load avg. on nightshade is WARNING: WARNING - load average: 14.79, 14.99, 15.36 [13:12:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:17:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 258913 MB (4% inode=55%): [13:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 934718.000000 [13:19:22] Load avg. on nightshade is OK: OK - load average: 13.70, 14.37, 14.97 [13:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122176 MB (20% inode=99%): [13:49:31] Free Memory on damiana is WARNING: WARNING - 6.2% (517932 kB) free! [13:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:01:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.88, 23.08, 22.54 [14:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:05:22] Load avg. on nightshade is WARNING: WARNING - load average: 17.44, 16.46, 14.93 [14:06:31] Free Memory on damiana is OK: OK - 7.1% (594980 kB) free. [14:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:09:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 258254 MB (4% inode=55%): [14:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 938318.000000 [14:21:22] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 75329 MB (7% inode=99%): [14:33:30] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (383388 kB) free! [14:34:31] Free Memory on damiana is WARNING: WARNING - 5.1% (425304 kB) free! [14:37:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121921 MB (20% inode=99%): [14:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:50:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:56:31] Free Memory on damiana is OK: OK - 7.4% (621148 kB) free. [15:01:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.49, 23.82, 22.41 [15:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:05:21] Load avg. on nightshade is WARNING: WARNING - load average: 13.84, 15.16, 15.71 [15:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:11:22] Load avg. on nightshade is OK: OK - load average: 12.15, 13.76, 14.93 [15:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 257648 MB (4% inode=55%): [15:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 941919.000000 [15:19:31] Free Memory on damiana is WARNING: WARNING - 5.1% (426784 kB) free! [15:20:31] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417776 kB) free! [15:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121709 MB (20% inode=99%): [15:47:31] Free Memory on damiana is WARNING: WARNING - 6.0% (505088 kB) free! [15:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:50:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:57:22] Load avg. on nightshade is WARNING: WARNING - load average: 21.14, 17.69, 15.44 [16:01:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.23, 23.93, 22.02 [16:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 256806 MB (4% inode=55%): [16:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 945518.000000 [16:20:21] Load avg. on nightshade is OK: OK - load average: 8.07, 11.74, 14.57 [16:28:31] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417808 kB) free! [16:29:30] Free Memory on damiana is WARNING: WARNING - 6.3% (529040 kB) free! [16:31:31] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420900 kB) free! [16:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [16:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121405 MB (20% inode=99%): [16:49:30] Load avg. on willow is WARNING: WARNING - load average: 19.21, 19.59, 19.95 [16:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:50:21] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:50:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.92, 22.19, 20.81 [16:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:57:31] Free Memory on damiana is WARNING: WARNING - 6.7% (561672 kB) free! [16:58:00] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:06:31] Free Memory on damiana is OK: OK - 7.2% (599244 kB) free. [17:07:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:09:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 256180 MB (4% inode=55%): [17:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 949118.000000 [17:19:31] Free Memory on damiana is WARNING: WARNING - 5.9% (494044 kB) free! [17:22:31] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422244 kB) free! [17:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121097 MB (19% inode=99%): [17:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:50:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:50:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.28, 20.67, 20.74 [17:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:58:01] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:59:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 255130 MB (4% inode=55%): [18:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 952718.000000 [18:18:30] Free Memory on damiana is WARNING: WARNING - 6.2% (518132 kB) free! [18:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:40:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120797 MB (19% inode=99%): [18:48:32] Free Memory on damiana is OK: OK - 7.1% (592160 kB) free. [18:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:50:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:50:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.73, 22.31, 21.15 [18:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:58:00] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:59:40] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:02:01] /sql on cassia is OK: DISK OK - free space: /sql 130911 MB (11% inode=99%): [19:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:03:31] Free Memory on damiana is WARNING: WARNING - 5.3% (445716 kB) free! [19:07:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:09:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129778 MB (10% inode=99%): [19:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:16:31] Free Memory on damiana is OK: OK - 7.1% (594908 kB) free. [19:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 253523 MB (4% inode=55%): [19:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 956319.000000 [19:24:31] Free Memory on damiana is WARNING: WARNING - 7.0% (586836 kB) free! [19:29:31] Load avg. on willow is WARNING: WARNING - load average: 17.98, 19.07, 19.88 [19:30:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.07, 20.30, 20.25 [19:33:30] Load avg. on willow is WARNING: WARNING - load average: 18.34, 19.39, 19.90 [19:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120499 MB (19% inode=99%): [19:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:50:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:58:01] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:59:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:09:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:12:31] Free Memory on damiana is WARNING: WARNING - 6.4% (537920 kB) free! [20:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 252032 MB (4% inode=55%): [20:17:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.63, 20.68, 20.71 [20:18:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 959918.000000 [20:27:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (408128 kB) free! [20:29:31] Free Memory on damiana is WARNING: WARNING - 5.2% (438768 kB) free! [20:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120199 MB (19% inode=99%): [20:48:30] Free Memory on damiana is OK: OK - 7.1% (592900 kB) free. [20:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:50:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:58:01] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:59:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:12:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 251380 MB (4% inode=55%): [21:17:32] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.32, 21.02, 21.43 [21:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 963518.000000 [21:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:33:30] Free Memory on damiana is WARNING: WARNING - 6.2% (519820 kB) free! [21:37:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [21:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119963 MB (19% inode=99%): [21:46:31] Free Memory on damiana is OK: OK - 7.1% (593628 kB) free. [21:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:50:21] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:58:01] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:59:31] Load avg. on willow is WARNING: WARNING - load average: 18.97, 19.32, 19.98 [21:59:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:00:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.73, 20.65, 20.39 [22:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:08:31] Free Memory on damiana is WARNING: WARNING - 5.7% (476560 kB) free! [22:09:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:09:31] Load avg. on willow is WARNING: WARNING - load average: 18.58, 19.39, 19.94 [22:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 250843 MB (4% inode=55%): [22:18:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 967118.000000 [22:23:31] Free Memory on damiana is OK: OK - 7.1% (591700 kB) free. [22:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:32:31] Free Memory on damiana is WARNING: WARNING - 5.3% (444296 kB) free! [22:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:39:32] Free Memory on damiana is OK: OK - 7.1% (598228 kB) free. [22:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119705 MB (19% inode=99%): [22:44:01] /sql on cassia is OK: DISK OK - free space: /sql 132185 MB (11% inode=99%): [22:47:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.38, 19.90, 20.09 [22:49:31] Load avg. on willow is WARNING: WARNING - load average: 19.29, 19.68, 19.98 [22:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:50:22] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:50:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.29, 20.28, 20.16 [22:51:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129424 MB (10% inode=99%): [22:54:31] Load avg. on willow is WARNING: WARNING - load average: 19.37, 19.76, 19.97 [22:56:31] Free Memory on damiana is WARNING: WARNING - 5.5% (465068 kB) free! [22:58:01] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:59:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:10:31] Free Memory on damiana is OK: OK - 7.2% (601332 kB) free. [23:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 248996 MB (4% inode=55%): [23:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 970718.000000 [23:18:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.95, 21.14, 21.17 [23:24:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:33:31] Free Memory on damiana is WARNING: WARNING - 6.7% (562096 kB) free! [23:34:31] Free Memory on damiana is OK: OK - 7.3% (609752 kB) free. [23:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119488 MB (19% inode=99%): [23:44:31] Free Memory on damiana is WARNING: WARNING - 5.8% (482268 kB) free! [23:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:50:21] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:58:00] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:59:41] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates).