[00:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:06:33] Load avg. on willow is WARNING: WARNING - load average: 15.39, 16.39, 15.73 [00:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 303715 MB (5% inode=55%): [00:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 715029.000000 [00:33:54] Free Memory on damiana is WARNING: WARNING - 5.8% (483612 kB) free! [00:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:37:54] Free Memory on damiana is OK: OK - 7.3% (609340 kB) free. [00:41:14] nacht ts [00:42:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:47:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:49:54] Free Memory on damiana is WARNING: WARNING - 6.4% (535508 kB) free! [00:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:58:54] Free Memory on damiana is OK: OK - 7.1% (594156 kB) free. [01:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.75, 21.38, 18.51 [01:01:33] Load avg. on willow is WARNING: WARNING - load average: 25.20, 20.98, 18.55 [01:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 306339 MB (5% inode=55%): [01:16:05] Load avg. on ortelius is WARNING: WARNING - load average: 18.06, 17.07, 10.78 [01:16:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 718620.000000 [01:18:05] Load avg. on ortelius is OK: OK - load average: 9.94, 14.53, 10.62 [01:25:04] Load avg. on ortelius is WARNING: WARNING - load average: 14.09, 18.07, 14.14 [01:30:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.61, 20.83, 19.14 [01:31:34] Load avg. on willow is WARNING: WARNING - load average: 23.17, 20.35, 19.08 [01:31:54] Free Memory on damiana is WARNING: WARNING - 6.1% (513140 kB) free! [01:32:54] Free Memory on damiana is OK: OK - 7.5% (625348 kB) free. [01:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:42:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:46:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:47:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:53:04] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 26.37, 26.77, 24.78 [01:53:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.01, 22.03, 19.34 [02:01:33] Load avg. on willow is WARNING: WARNING - load average: 25.26, 21.50, 19.32 [02:03:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:09:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:14:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 305947 MB (5% inode=55%): [02:16:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 722219.000000 [02:20:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 79079 MB (8% inode=99%): [02:22:59] Free Memory on damiana is WARNING: WARNING - 5.2% (434700 kB) free! [02:27:59] Free Memory on damiana is CRITICAL: CRITICAL - 3.9% (330888 kB) free! [02:29:58] Free Memory on damiana is WARNING: WARNING - 6.1% (508912 kB) free! [02:35:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:42:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:46:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:47:29] Load avg. on ortelius is WARNING: WARNING - load average: 9.87, 14.68, 19.77 [02:47:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:51:30] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 27.74, 18.91, 20.06 [02:53:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:55:29] Load avg. on ortelius is WARNING: WARNING - load average: 11.33, 17.55, 19.55 [03:00:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.03, 20.58, 18.86 [03:01:39] Load avg. on willow is WARNING: WARNING - load average: 24.68, 20.60, 18.98 [03:03:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:06:29] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 27.55, 22.70, 20.50 [03:09:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:11:29] Load avg. on ortelius is WARNING: WARNING - load average: 7.41, 18.45, 19.81 [03:14:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 302390 MB (5% inode=55%): [03:16:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 725819.000000 [03:20:29] Load avg. on ortelius is OK: OK - load average: 4.61, 8.98, 14.45 [03:35:30] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:40:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.56, 20.59, 20.05 [03:41:39] Load avg. on willow is WARNING: WARNING - load average: 20.07, 20.14, 19.93 [03:42:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:44:00] Free Memory on damiana is WARNING: WARNING - 5.7% (481136 kB) free! [03:44:59] Free Memory on damiana is OK: OK - 8.6% (719844 kB) free. [03:46:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:47:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:53:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:00:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.03, 21.73, 20.21 [04:03:39] Load avg. on willow is WARNING: WARNING - load average: 19.32, 20.46, 19.98 [04:04:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:05:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.14, 21.42, 20.37 [04:10:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:14:28] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128703 MB (10% inode=99%): [04:15:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 300652 MB (5% inode=55%): [04:17:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 729484.000000 [04:36:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:43:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:47:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:47:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:57:38] Load avg. on willow is WARNING: WARNING - load average: 16.94, 17.54, 18.69 [05:00:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.41, 22.66, 20.26 [05:03:38] Load avg. on willow is WARNING: WARNING - load average: 18.71, 20.57, 19.86 [05:04:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:10:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:15:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 301828 MB (5% inode=55%): [05:15:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.41, 21.32, 20.25 [05:17:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 733084.000000 [05:23:29] /sql on cassia is OK: DISK OK - free space: /sql 133698 MB (11% inode=99%): [05:23:38] Load avg. on willow is WARNING: WARNING - load average: 17.23, 19.40, 19.83 [05:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:36:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:43:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:48:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:48:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:49:28] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126832 MB (10% inode=99%): [05:50:29] /sql on cassia is OK: DISK OK - free space: /sql 131276 MB (11% inode=99%): [06:00:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.25, 21.25, 19.32 [06:01:38] Load avg. on willow is WARNING: WARNING - load average: 24.39, 20.93, 19.33 [06:04:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:10:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:15:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 301436 MB (5% inode=55%): [06:17:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 736683.000000 [06:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:33:08] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (416888 kB) free! [06:34:08] Free Memory on damiana is WARNING: WARNING - 5.4% (449960 kB) free! [06:36:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:37:08] Free Memory on damiana is OK: OK - 7.2% (599336 kB) free. [06:43:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:48:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:01:38] Load avg. on willow is WARNING: WARNING - load average: 19.81, 18.43, 18.10 [07:04:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:10:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:15:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 301158 MB (5% inode=55%): [07:17:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740284.000000 [07:23:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:36:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:43:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:48:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:48:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:01:38] Load avg. on willow is WARNING: WARNING - load average: 22.15, 19.30, 17.83 [08:04:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:10:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:15:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 300831 MB (5% inode=55%): [08:17:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743884.000000 [08:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:36:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:43:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:48:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:48:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:01:38] Load avg. on willow is WARNING: WARNING - load average: 20.20, 18.21, 17.48 [09:04:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:10:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:15:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 300031 MB (5% inode=55%): [09:17:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 747484.000000 [09:23:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:30:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.81, 20.30, 18.40 [09:31:38] Load avg. on willow is WARNING: WARNING - load average: 22.28, 19.79, 18.34 [09:36:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:43:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:48:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:48:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:59:51] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2808 MB (14% inode=99%): [10:03:11] Free Memory on damiana is WARNING: WARNING - 5.4% (449804 kB) free! [10:04:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:04:52] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 229 MB (1% inode=99%): [10:07:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2861 MB (14% inode=99%): [10:10:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:11:55] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5156 MB (17% inode=92%): [10:11:55] / on wolfsbane is WARNING: DISK WARNING - free space: / 5150 MB (17% inode=92%): [10:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 297975 MB (5% inode=55%): [10:17:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 751093.000000 [10:28:14] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (363148 kB) free! [10:30:13] Free Memory on damiana is WARNING: WARNING - 5.8% (482040 kB) free! [10:31:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (391836 kB) free! [10:31:51] Load avg. on willow is WARNING: WARNING - load average: 20.54, 19.40, 18.67 [10:31:51] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1929 MB (10% inode=99%): [10:32:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3407 MB (16% inode=99%): [10:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:43:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:48:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:00:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.22, 21.18, 19.14 [11:01:51] Load avg. on willow is WARNING: WARNING - load average: 22.39, 20.60, 19.07 [11:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:10:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:11:55] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5098 MB (17% inode=92%): [11:11:55] / on wolfsbane is WARNING: DISK WARNING - free space: / 5098 MB (17% inode=92%): [11:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 297241 MB (5% inode=55%): [11:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 754693.000000 [11:32:51] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2678 MB (13% inode=99%): [11:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:43:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:48:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:48:52] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 774 MB (4% inode=99%): [11:49:51] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2521 MB (12% inode=99%): [11:52:51] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 2025 MB (10% inode=99%): [11:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:00:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.92, 22.57, 19.55 [12:01:51] Load avg. on willow is WARNING: WARNING - load average: 25.71, 22.11, 19.59 [12:02:52] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2510 MB (8% inode=92%): [12:02:52] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2510 MB (8% inode=92%): [12:03:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5082 MB (16% inode=92%): [12:03:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 5082 MB (16% inode=92%): [12:04:11] Free Memory on damiana is WARNING: WARNING - 6.7% (562376 kB) free! [12:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:08:11] Free Memory on damiana is OK: OK - 7.3% (608668 kB) free. [12:10:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:10:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.99, 22.86, 20.56 [12:11:51] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129429 MB (10% inode=99%): [12:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 296592 MB (5% inode=55%): [12:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 758292.000000 [12:23:11] Free Memory on damiana is WARNING: WARNING - 6.6% (556884 kB) free! [12:28:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (376156 kB) free! [12:28:51] Load avg. on willow is WARNING: WARNING - load average: 18.50, 19.27, 19.91 [12:30:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.13, 22.52, 20.99 [12:30:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2168 MB (11% inode=99%): [12:35:52] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1026 MB (5% inode=99%): [12:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:38:51] Load avg. on willow is WARNING: WARNING - load average: 17.77, 19.14, 19.92 [12:43:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:03:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5049 MB (16% inode=92%): [13:03:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 5048 MB (16% inode=92%): [13:04:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:10:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:10:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.20, 21.12, 20.60 [13:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 295804 MB (5% inode=55%): [13:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 761893.000000 [13:23:52] Load avg. on willow is WARNING: WARNING - load average: 18.30, 19.32, 19.96 [13:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:30:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.42, 20.38, 20.03 [13:32:52] Load avg. on willow is WARNING: WARNING - load average: 20.30, 20.16, 19.99 [13:35:51] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 189 MB (0% inode=99%): [13:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:41:52] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:43:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:43:52] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:44:01] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 188 MB (0% inode=99%): [13:44:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:45:01] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:46:51] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 9.197 second response time [13:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:52:51] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.030 second response time [13:55:02] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:03:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4992 MB (16% inode=92%): [14:03:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 4992 MB (16% inode=92%): [14:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:10:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.84, 21.01, 20.51 [14:14:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:14:52] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1505 MB (7% inode=99%): [14:15:04] DaB. * [Toolserver-announce] Split of s2/s5 (cassia) [14:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 295317 MB (5% inode=55%): [14:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 765493.000000 [14:20:51] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 78921 MB (8% inode=99%): [14:21:52] Load avg. on willow is WARNING: WARNING - load average: 18.46, 19.37, 19.96 [14:23:11] Free Memory on damiana is WARNING: WARNING - 6.8% (573520 kB) free! [14:24:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:28:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (397188 kB) free! [14:28:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [14:30:11] Free Memory on damiana is WARNING: WARNING - 5.9% (495236 kB) free! [14:30:52] /sql on cassia is OK: DISK OK - free space: /sql 131962 MB (11% inode=99%): [14:31:11] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422144 kB) free! [14:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:38:51] MySQL slave on z-dat-s3-a is OK: Uptime: 1330893 Threads: 19 Questions: 1267962254 Slow queries: 73000 Opens: 10233927 Flush tables: 1 Open tables: 16384 Queries per second avg: 952.715 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1003 [14:40:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.04, 20.86, 20.08 [14:41:52] Load avg. on willow is WARNING: WARNING - load average: 20.36, 20.30, 19.93 [14:43:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:02:10] Free Memory on damiana is WARNING: WARNING - 5.9% (496848 kB) free! [15:03:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4922 MB (16% inode=92%): [15:03:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 4919 MB (16% inode=92%): [15:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:05:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.71, 20.42, 20.01 [15:06:54] Load avg. on willow is WARNING: WARNING - load average: 19.20, 20.05, 19.91 [15:10:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.86, 21.07, 20.25 [15:14:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294593 MB (5% inode=55%): [15:15:51] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1298 MB (6% inode=99%): [15:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 769093.000000 [15:23:11] Free Memory on damiana is OK: OK - 7.1% (591388 kB) free. [15:23:52] Load avg. on willow is WARNING: WARNING - load average: 18.79, 19.67, 20.00 [15:30:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.61, 20.83, 20.16 [15:33:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (353916 kB) free! [15:34:17] Free Memory on damiana is WARNING: WARNING - 5.5% (461540 kB) free! [15:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:36:51] Load avg. on willow is WARNING: WARNING - load average: 18.87, 19.75, 19.92 [15:43:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:03:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4891 MB (16% inode=92%): [16:03:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 4891 MB (16% inode=92%): [16:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:14:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 294095 MB (5% inode=55%): [16:15:51] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1392 MB (6% inode=99%): [16:16:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.11, 19.95, 20.43 [16:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 772693.000000 [16:21:11] Free Memory on damiana is OK: OK - 7.1% (593552 kB) free. [16:23:52] Load avg. on willow is WARNING: WARNING - load average: 18.15, 19.14, 19.89 [16:26:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.76, 20.58, 20.27 [16:31:11] Free Memory on damiana is WARNING: WARNING - 5.1% (427556 kB) free! [16:32:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (413740 kB) free! [16:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:37:52] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129368 MB (10% inode=99%): [16:43:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:46:52] /sql on cassia is OK: DISK OK - free space: /sql 130941 MB (11% inode=99%): [16:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:54:51] Load avg. on willow is WARNING: WARNING - load average: 17.41, 18.36, 19.92 [17:00:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.52, 22.17, 20.77 [17:03:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4814 MB (16% inode=92%): [17:03:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 4814 MB (16% inode=92%): [17:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:08:51] Load avg. on willow is WARNING: WARNING - load average: 17.34, 19.14, 19.89 [17:12:11] Free Memory on damiana is WARNING: WARNING - 5.4% (451348 kB) free! [17:13:11] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421128 kB) free! [17:14:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293443 MB (5% inode=55%): [17:15:11] Free Memory on damiana is WARNING: WARNING - 5.2% (436392 kB) free! [17:15:51] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 840 MB (4% inode=99%): [17:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 776293.000000 [17:26:51] /sql on cassia is WARNING: DISK WARNING - free space: /sql 127194 MB (10% inode=99%): [17:28:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (346068 kB) free! [17:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [17:40:51] Load avg. on willow is WARNING: WARNING - load average: 20.45, 19.14, 19.32 [17:43:51] /sql on cassia is OK: DISK OK - free space: /sql 131313 MB (11% inode=99%): [17:43:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:01:21] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:01:51] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 7.419 second response time [18:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:03:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4784 MB (15% inode=92%): [18:03:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 4784 MB (15% inode=92%): [18:04:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:07:20] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:07:52] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.044 second response time [18:07:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292896 MB (5% inode=55%): [18:15:51] /tmp on ortelius is OK: DISK OK - free space: /tmp 19717 MB (96% inode=99%): [18:16:52] /tmp on wolfsbane is OK: DISK OK - free space: / 16241 MB (54% inode=92%): [18:16:52] / on wolfsbane is OK: DISK OK - free space: / 16244 MB (54% inode=92%): [18:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 779893.000000 [18:25:52] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129500 MB (10% inode=99%): [18:26:52] /sql on cassia is OK: DISK OK - free space: /sql 130459 MB (11% inode=99%): [18:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:40:51] Load avg. on willow is WARNING: WARNING - load average: 22.27, 19.48, 19.11 [18:41:04] Johannes Kroll * [Toolserver-l] Funny /tmp fail [18:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:51:16] Free Memory on damiana is WARNING: WARNING - 5.1% (425748 kB) free! [18:52:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (411804 kB) free! [18:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:54:11] Free Memory on damiana is WARNING: WARNING - 5.1% (430444 kB) free! [19:02:52] /sql on cassia is WARNING: DISK WARNING - free space: /sql 128353 MB (10% inode=99%): [19:02:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:04:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:08:51] /sql on cassia is OK: DISK OK - free space: /sql 131758 MB (11% inode=99%): [19:08:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:14:11] Free Memory on damiana is OK: OK - 7.3% (608916 kB) free. [19:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292342 MB (5% inode=55%): [19:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 783494.000000 [19:21:11] Free Memory on damiana is WARNING: WARNING - 5.9% (495344 kB) free! [19:28:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (334004 kB) free! [19:30:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.18, 22.28, 19.80 [19:31:51] Load avg. on willow is WARNING: WARNING - load average: 22.97, 21.43, 19.66 [19:35:51] /sql on cassia is WARNING: DISK WARNING - free space: /sql 130061 MB (10% inode=99%): [19:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:37:51] /sql on cassia is OK: DISK OK - free space: /sql 130610 MB (11% inode=99%): [19:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:51:11] Free Memory on damiana is WARNING: WARNING - 6.7% (560980 kB) free! [19:53:51] /sql on cassia is WARNING: DISK WARNING - free space: /sql 127053 MB (10% inode=99%): [20:02:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:08:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:09:11] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (407928 kB) free! [20:10:04] DaB. * Re: [Toolserver-l] Funny /tmp fail [20:10:52] /sql on cassia is OK: DISK OK - free space: /sql 130565 MB (11% inode=99%): [20:12:11] Free Memory on damiana is WARNING: WARNING - 5.1% (431232 kB) free! [20:15:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291525 MB (5% inode=55%): [20:15:12] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (391688 kB) free! [20:17:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 787094.000000 [20:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:31:52] Load avg. on willow is WARNING: WARNING - load average: 19.79, 18.99, 18.63 [20:36:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:40:30] Reedy: according to toolserver nlwikivoyage_p.site_stats there are 18 * 10^(18) articles. is this an replication failure or sth wrong with the value on master? [20:48:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:53:04] Johannes Kroll * Re: [Toolserver-l] Funny /tmp fail [21:02:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:04:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:08:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291075 MB (5% inode=55%): [21:17:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 790693.000000 [21:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:31:52] Load avg. on willow is WARNING: WARNING - load average: 21.84, 20.23, 19.29 [21:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:48:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:02:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:04:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:08:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:10:02] hello all [22:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290422 MB (5% inode=55%): [22:17:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 794293.000000 [22:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:30:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.49, 22.20, 19.63 [22:31:51] Load avg. on willow is WARNING: WARNING - load average: 23.14, 21.30, 19.48 [22:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:42:04] Ohai. Who do I talk to to reactivate an existing account? Well, if it still exists. [22:48:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:49:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:02:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:10:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:10:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:15:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289956 MB (5% inode=55%): [23:17:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 797894.000000 [23:24:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:31:51] Load avg. on willow is WARNING: WARNING - load average: 20.70, 19.98, 19.23 [23:39:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:50:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:50:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default