[00:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:05:30] Free Memory on damiana is WARNING: WARNING - 5.4% (455960 kB) free! [00:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:09:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:11:30] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421668 kB) free! [00:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:12:31] Free Memory on damiana is WARNING: WARNING - 5.2% (433012 kB) free! [00:13:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 226292 MB (4% inode=54%): [00:17:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.58, 24.15, 23.83 [00:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1060719.000000 [00:23:21] Load avg. on nightshade is WARNING: WARNING - load average: 17.86, 17.73, 17.62 [00:27:31] Free Memory on damiana is CRITICAL: CRITICAL - 3.5% (294572 kB) free! [00:32:03] DaB. * Re: [Toolserver-l] Cron on submit [00:32:58] nacht ts [00:34:09] apmon_: I would suggest to start with the postGres, but the OSM-people should say how they like it – I just provide the server :) [00:34:12] cu [00:34:31] Free Memory on damiana is WARNING: WARNING - 5.2% (433144 kB) free! [00:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 114121 MB (18% inode=99%): [00:45:31] Free Memory on damiana is OK: OK - 7.2% (602212 kB) free. [00:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:07:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:13:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228920 MB (4% inode=54%): [01:17:32] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.00, 23.51, 23.68 [01:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1064320.000000 [01:23:21] Load avg. on nightshade is WARNING: WARNING - load average: 17.80, 17.99, 17.82 [01:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113862 MB (18% inode=99%): [01:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:09:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:13:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228696 MB (4% inode=54%): [02:17:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.51, 25.77, 25.75 [02:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1067919.000000 [02:21:21] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 76598 MB (7% inode=99%): [02:23:21] Load avg. on nightshade is WARNING: WARNING - load average: 18.35, 18.21, 17.92 [02:33:31] Free Memory on damiana is WARNING: WARNING - 6.4% (534956 kB) free! [02:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:38:31] Free Memory on damiana is OK: OK - 7.2% (601624 kB) free. [02:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113678 MB (18% inode=99%): [02:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:12:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:13:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:17:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228731 MB (4% inode=54%): [03:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.28, 29.55, 27.83 [03:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1071520.000000 [03:23:21] Load avg. on nightshade is WARNING: WARNING - load average: 17.44, 17.35, 17.56 [03:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:39:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (383192 kB) free! [03:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113564 MB (18% inode=99%): [03:47:31] Free Memory on damiana is WARNING: WARNING - 5.1% (429872 kB) free! [03:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:04:31] Free Memory on damiana is OK: OK - 7.4% (619848 kB) free. [04:07:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:09:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:14:31] Free Memory on damiana is WARNING: WARNING - 5.6% (470460 kB) free! [04:15:31] Free Memory on damiana is OK: OK - 7.7% (644668 kB) free. [04:17:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.51, 29.37, 28.59 [04:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1075119.000000 [04:18:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 229478 MB (4% inode=54%): [04:23:21] Load avg. on nightshade is WARNING: WARNING - load average: 17.89, 18.17, 18.03 [04:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113483 MB (18% inode=99%): [04:47:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402788 kB) free! [04:49:31] Free Memory on damiana is WARNING: WARNING - 5.2% (436152 kB) free! [04:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:51:31] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (417348 kB) free! [05:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:09:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.47, 30.02, 30.21 [05:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1078720.000000 [05:18:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 229536 MB (4% inode=54%): [05:23:21] Load avg. on nightshade is WARNING: WARNING - load average: 18.53, 18.71, 18.67 [05:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:30:01] /sql on cassia is OK: DISK OK - free space: /sql 132744 MB (11% inode=99%): [05:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113380 MB (18% inode=99%): [05:42:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126392 MB (10% inode=99%): [05:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:10:31] Free Memory on damiana is WARNING: WARNING - 6.9% (575612 kB) free! [06:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.14, 28.55, 29.32 [06:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1082320.000000 [06:18:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228859 MB (4% inode=54%): [06:23:21] Load avg. on nightshade is WARNING: WARNING - load average: 17.97, 18.36, 18.27 [06:28:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (345756 kB) free! [06:29:31] Free Memory on damiana is WARNING: WARNING - 5.2% (437536 kB) free! [06:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 113134 MB (18% inode=99%): [06:40:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (354680 kB) free! [06:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:12:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 23.56, 21.82, 20.07 [07:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.14, 24.68, 26.31 [07:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1085919.000000 [07:18:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 228669 MB (4% inode=54%): [07:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112926 MB (18% inode=99%): [07:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:12:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 29.05, 24.83, 23.66 [08:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.87, 27.16, 26.61 [08:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1089520.000000 [08:18:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 227227 MB (4% inode=54%): [08:20:00] /sql on cassia is OK: DISK OK - free space: /sql 130163 MB (11% inode=99%): [08:27:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128287 MB (10% inode=99%): [08:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 110976 MB (18% inode=99%): [08:40:01] /sql on cassia is OK: DISK OK - free space: /sql 130980 MB (11% inode=99%): [08:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:06:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129175 MB (10% inode=99%): [09:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:12:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 24.55, 25.37, 24.87 [09:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.42, 26.89, 26.86 [09:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1093120.000000 [09:18:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 226671 MB (4% inode=54%): [09:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112245 MB (18% inode=99%): [09:44:01] /sql on cassia is OK: DISK OK - free space: /sql 130845 MB (11% inode=99%): [09:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:51:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129384 MB (10% inode=99%): [10:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:12:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 24.27, 24.00, 24.74 [10:13:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.30, 25.00, 25.75 [10:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1096720.000000 [10:18:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 226057 MB (4% inode=54%): [10:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:40:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 112124 MB (18% inode=99%): [10:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:12:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:12:22] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 25.30, 25.61, 24.88 [11:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.28, 25.93, 25.63 [11:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1100320.000000 [11:18:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 225422 MB (4% inode=54%): [11:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111861 MB (18% inode=99%): [11:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:12:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 25.47, 24.35, 24.32 [12:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.29, 28.80, 27.47 [12:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1103919.000000 [12:18:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 224559 MB (4% inode=53%): [12:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111636 MB (18% inode=99%): [12:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:12:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:12:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 23.22, 25.45, 26.26 [13:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.15, 26.10, 26.18 [13:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1107519.000000 [13:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 223785 MB (4% inode=53%): [13:20:04] Silke Meyer * [Toolserver-l] Spec for Tool Labs [13:20:04] Magnus Manske * [Toolserver-l] FYI: TS wikivoyage is missing a few pages [13:37:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111399 MB (18% inode=99%): [13:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:07:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:09:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:12:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:12:22] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 23.42, 23.42, 23.21 [14:13:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.35, 25.81, 26.00 [14:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1111120.000000 [14:18:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 223096 MB (4% inode=53%): [14:21:21] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 76918 MB (7% inode=99%): [14:37:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:40:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 111134 MB (18% inode=99%): [14:50:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:03:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output