[00:27:39] Free Memory on damiana is WARNING: WARNING - 6.0% (501172 kB) free! [00:36:39] Free Memory on damiana is OK: OK - 7.1% (594824 kB) free. [00:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [00:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [00:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.24, 27.81, 28.79 [00:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:45:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 440897 MB (8% inode=39%): [00:45:21] Sun Grid Engine execd on willow is WARNING: (null) [00:45:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:50:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126749 MB (20% inode=99%): [00:50:51] Sun Grid Engine execd on yarrow is WARNING: (null) [00:51:20] Sun Grid Engine execd on nightshade is WARNING: (null) [00:56:39] Free Memory on damiana is WARNING: WARNING - 6.5% (547652 kB) free! [00:57:14] DaBPunkt: can you explain my quota -v output? [00:57:49] how is the output? [00:58:38] Filesystem usage quota limit timeleft files quota limit timeleft [00:58:48] /home 285.6M 256.0M 384.0M 6.8 days 1321 0 0 [00:59:31] it means that you use too much space and have 6.8 days left to fix this [00:59:42] what will happen if i don't? [00:59:58] can't add any more to your account. [01:00:09] Only lets you remove data, not add data. [01:00:30] and if i try i will empty files :\ [01:03:08] you can request a extend of the quota [01:04:42] i move files to user-store [01:04:54] and maybe i can delete some old stuff [01:28:30] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 93375 MB (9% inode=99%): [01:32:40] Free Memory on damiana is WARNING: WARNING - 6.5% (546728 kB) free! [01:34:40] Free Memory on damiana is OK: OK - 7.1% (597648 kB) free. [01:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [01:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [01:44:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.02, 31.94, 31.76 [01:44:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 439736 MB (8% inode=39%): [01:45:21] Sun Grid Engine execd on willow is WARNING: (null) [01:45:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:50:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124878 MB (20% inode=99%): [01:50:51] Sun Grid Engine execd on yarrow is WARNING: (null) [01:51:21] Sun Grid Engine execd on nightshade is WARNING: (null) [02:04:59] /sql on ptolemy is OK: DISK OK - free space: /sql 127832 MB (21% inode=99%): [02:13:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126904 MB (20% inode=99%): [02:15:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:21:53] nacht ts [02:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [02:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [02:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.79, 31.92, 31.05 [02:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [02:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 438139 MB (8% inode=39%): [02:45:20] Sun Grid Engine execd on willow is WARNING: (null) [02:45:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:50:50] Sun Grid Engine execd on yarrow is WARNING: (null) [02:51:20] Sun Grid Engine execd on nightshade is WARNING: (null) [03:15:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126577 MB (20% inode=99%): [03:39:40] Free Memory on damiana is WARNING: WARNING - 6.5% (542952 kB) free! [03:44:31] Sun Grid Engine execd on wolfsbane is WARNING: (null) [03:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [03:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.64, 37.09, 36.16 [03:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [03:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 435118 MB (8% inode=39%): [03:45:20] Sun Grid Engine execd on willow is WARNING: (null) [03:45:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:49:40] Free Memory on damiana is OK: OK - 7.1% (595624 kB) free. [03:50:50] Sun Grid Engine execd on yarrow is WARNING: (null) [03:51:20] Sun Grid Engine execd on nightshade is WARNING: (null) [04:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126551 MB (20% inode=99%): [04:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [04:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [04:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.85, 34.97, 34.88 [04:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:45:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [04:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434923 MB (8% inode=39%): [04:45:20] Sun Grid Engine execd on willow is WARNING: (null) [04:45:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:50:50] Sun Grid Engine execd on yarrow is WARNING: (null) [04:51:21] Sun Grid Engine execd on nightshade is WARNING: (null) [05:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126482 MB (20% inode=99%): [05:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [05:44:39] Sun Grid Engine execd on ortelius is WARNING: (null) [05:44:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.01, 35.65, 35.28 [05:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:45:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [05:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434743 MB (8% inode=39%): [05:45:20] Sun Grid Engine execd on willow is WARNING: (null) [05:45:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:50:50] Sun Grid Engine execd on yarrow is WARNING: (null) [05:51:20] Sun Grid Engine execd on nightshade is WARNING: (null) [06:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126445 MB (20% inode=99%): [06:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [06:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [06:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.20, 31.63, 31.96 [06:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:45:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:45:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [06:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434655 MB (8% inode=38%): [06:45:20] Sun Grid Engine execd on willow is WARNING: (null) [06:45:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:50:50] Sun Grid Engine execd on yarrow is WARNING: (null) [06:51:20] Sun Grid Engine execd on nightshade is WARNING: (null) [07:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126380 MB (20% inode=99%): [07:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [07:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [07:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.54, 32.13, 31.78 [07:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:45:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:45:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434593 MB (8% inode=38%): [07:45:20] Sun Grid Engine execd on willow is WARNING: (null) [07:45:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:50:50] Sun Grid Engine execd on yarrow is WARNING: (null) [07:51:20] Sun Grid Engine execd on nightshade is WARNING: (null) [08:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126281 MB (20% inode=99%): [08:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [08:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [08:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.28, 32.45, 32.85 [08:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:45:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434535 MB (8% inode=38%): [08:45:21] Sun Grid Engine execd on willow is WARNING: (null) [08:45:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:50:50] Sun Grid Engine execd on yarrow is WARNING: (null) [08:51:20] Sun Grid Engine execd on nightshade is WARNING: (null) [09:15:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126203 MB (20% inode=99%): [09:44:30] Sun Grid Engine execd on wolfsbane is WARNING: (null) [09:44:40] Sun Grid Engine execd on ortelius is WARNING: (null) [09:44:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.99, 31.62, 31.73 [09:44:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:45:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:45:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434442 MB (8% inode=38%): [09:45:21] Sun Grid Engine execd on willow is WARNING: (null) [09:45:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:51:07] Sun Grid Engine execd on yarrow is WARNING: (null) [09:51:08] / on ptolemy is WARNING: DISK WARNING - free space: / 4175 MB (20% inode=91%): [09:52:17] Sun Grid Engine execd on nightshade is WARNING: (null) [09:59:07] Free Memory on damiana is WARNING: WARNING - 6.5% (543204 kB) free! [10:08:08] Free Memory on damiana is OK: OK - 7.1% (594760 kB) free. [10:15:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:19:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126108 MB (20% inode=99%): [10:27:08] /sql on cassia is WARNING: DISK WARNING - free space: /sql 101609 MB (8% inode=99%): [10:38:08] Free Memory on damiana is WARNING: WARNING - 6.9% (581284 kB) free! [10:45:08] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.61, 31.10, 31.21 [10:45:08] Sun Grid Engine execd on ortelius is WARNING: (null) [10:45:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:45:27] Sun Grid Engine execd on wolfsbane is WARNING: (null) [10:46:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:46:08] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:46:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434160 MB (8% inode=38%): [10:46:18] Sun Grid Engine execd on willow is WARNING: (null) [10:47:07] Free Memory on damiana is OK: OK - 8.0% (671352 kB) free. [10:51:07] Sun Grid Engine execd on yarrow is WARNING: (null) [10:52:08] / on ptolemy is WARNING: DISK WARNING - free space: / 4011 MB (20% inode=91%): [10:52:17] Sun Grid Engine execd on nightshade is WARNING: (null) [11:01:08] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (358548 kB) free! [11:06:07] Free Memory on damiana is WARNING: WARNING - 5.1% (424612 kB) free! [11:15:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:19:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125995 MB (20% inode=99%): [11:45:08] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.05, 31.43, 31.09 [11:45:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:45:27] Sun Grid Engine execd on wolfsbane is WARNING: (null) [11:46:07] Sun Grid Engine execd on ortelius is WARNING: (null) [11:46:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:46:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:46:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 433552 MB (8% inode=38%): [11:46:17] Sun Grid Engine execd on willow is WARNING: (null) [11:47:08] Free Memory on damiana is WARNING: WARNING - 6.6% (557116 kB) free! [11:49:08] Free Memory on damiana is OK: OK - 8.0% (667308 kB) free. [11:51:07] Sun Grid Engine execd on yarrow is WARNING: (null) [11:52:08] / on ptolemy is WARNING: DISK WARNING - free space: / 3827 MB (19% inode=91%): [11:52:17] Sun Grid Engine execd on nightshade is WARNING: (null) [12:15:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:19:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125891 MB (20% inode=99%): [12:43:07] Free Memory on damiana is WARNING: WARNING - 6.1% (511932 kB) free! [12:45:07] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.13, 33.23, 32.24 [12:45:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:45:26] Sun Grid Engine execd on wolfsbane is WARNING: (null) [12:46:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:46:08] Sun Grid Engine execd on ortelius is WARNING: (null) [12:46:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 432749 MB (8% inode=38%): [12:46:17] Sun Grid Engine execd on willow is WARNING: (null) [12:47:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [12:51:08] Sun Grid Engine execd on yarrow is WARNING: (null) [12:52:07] / on ptolemy is WARNING: DISK WARNING - free space: / 3640 MB (18% inode=91%): [12:52:17] Sun Grid Engine execd on nightshade is WARNING: (null) [13:06:07] Free Memory on damiana is OK: OK - 7.2% (603912 kB) free. [13:15:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:19:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125720 MB (20% inode=99%): [13:28:07] Free Memory on damiana is WARNING: WARNING - 6.4% (536556 kB) free! [13:29:27] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 91496 MB (9% inode=99%): [13:36:07] Free Memory on damiana is OK: OK - 7.2% (599344 kB) free. [13:45:07] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.98, 36.73, 37.26 [13:45:27] Sun Grid Engine execd on wolfsbane is WARNING: (null) [13:46:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:46:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:46:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 428598 MB (8% inode=38%): [13:46:17] Sun Grid Engine execd on willow is WARNING: (null) [13:47:08] Sun Grid Engine execd on ortelius is WARNING: (null) [13:47:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:51:07] Sun Grid Engine execd on yarrow is WARNING: (null) [13:52:08] / on ptolemy is WARNING: DISK WARNING - free space: / 3440 MB (17% inode=91%): [13:52:17] Sun Grid Engine execd on nightshade is WARNING: (null) [13:59:08] Free Memory on damiana is WARNING: WARNING - 6.6% (551452 kB) free! [14:00:07] Free Memory on damiana is OK: OK - 8.1% (679332 kB) free. [14:15:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:19:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125563 MB (20% inode=99%): [14:39:07] Free Memory on damiana is WARNING: WARNING - 5.9% (494872 kB) free! [14:45:08] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.62, 35.88, 35.15 [14:45:27] Sun Grid Engine execd on wolfsbane is WARNING: (null) [14:46:07] Free Memory on damiana is OK: OK - 8.2% (691028 kB) free. [14:46:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:46:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:46:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 412252 MB (7% inode=37%): [14:46:17] Sun Grid Engine execd on willow is WARNING: (null) [14:47:07] Sun Grid Engine execd on ortelius is WARNING: (null) [14:47:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:51:07] Sun Grid Engine execd on yarrow is WARNING: (null) [14:52:07] / on ptolemy is WARNING: DISK WARNING - free space: / 3222 MB (16% inode=91%): [14:52:17] Sun Grid Engine execd on nightshade is WARNING: (null) [15:03:07] Free Memory on damiana is WARNING: WARNING - 5.1% (427828 kB) free! [15:05:07] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (408120 kB) free! [15:15:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:19:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125412 MB (20% inode=99%): [15:25:08] Free Memory on damiana is WARNING: WARNING - 6.3% (532092 kB) free! [15:25:36] hello all [15:25:40] @replag [15:25:40] DaBPunkt: s1-rr-a-wd: 14h 48m 54s [-0.67 s/s]; s1-user-wd: 19h 26m 12s [-0.53 s/s]; s2-user-wd: 19h 42m 35s [-0.57 s/s]; s3-user: 1m 36s [+0.00 s/s]; s4-user-wd: 12h 5m 21s [-0.82 s/s]; s5-user-wd: 19h 42m 35s [-0.57 s/s]; s6-user-wd: 17h 55m 22s [-0.49 s/s]; s7-user-wd: 12h 55m 51s [-0.77 s/s] [15:31:07] Free Memory on damiana is OK: OK - 7.7% (644716 kB) free. [15:45:07] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.54, 35.99, 35.76 [15:45:27] Sun Grid Engine execd on wolfsbane is WARNING: (null) [15:46:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:46:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:46:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 423880 MB (8% inode=38%): [15:46:17] Sun Grid Engine execd on willow is WARNING: (null) [15:47:07] Sun Grid Engine execd on ortelius is WARNING: (null) [15:48:08] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:51:07] Sun Grid Engine execd on yarrow is WARNING: (null) [15:52:07] / on ptolemy is WARNING: DISK WARNING - free space: / 3026 MB (15% inode=91%): [15:52:17] Sun Grid Engine execd on nightshade is WARNING: (null) [15:54:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 424078 MB (8% inode=38%): [15:54:12] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [15:54:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:58:31] Free Memory on damiana is WARNING: WARNING - 6.9% (576860 kB) free! [16:01:31] Free Memory on damiana is OK: OK - 7.3% (609260 kB) free. [16:06:41] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1123 MB (5% inode=99%): [16:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125240 MB (20% inode=99%): [16:20:41] /tmp on ortelius is OK: DISK OK - free space: /tmp 14002 MB (75% inode=99%): [16:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:33:32] Free Memory on damiana is WARNING: WARNING - 5.6% (466980 kB) free! [16:46:31] Free Memory on damiana is OK: OK - 7.4% (619700 kB) free. [16:53:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.52, 35.46, 35.41 [16:53:41] / on ptolemy is WARNING: DISK WARNING - free space: / 2824 MB (14% inode=91%): [16:53:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [16:54:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 422589 MB (7% inode=38%): [16:54:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125047 MB (20% inode=99%): [17:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:41:31] Free Memory on damiana is WARNING: WARNING - 6.8% (566192 kB) free! [17:46:31] Free Memory on damiana is OK: OK - 7.2% (599936 kB) free. [17:53:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.86, 37.05, 37.54 [17:53:41] / on ptolemy is WARNING: DISK WARNING - free space: / 2640 MB (13% inode=91%): [17:53:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:54:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [17:54:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 421800 MB (7% inode=38%): [17:54:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:00:32] Free Memory on damiana is WARNING: WARNING - 6.6% (551208 kB) free! [18:14:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:15:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:17:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124848 MB (20% inode=99%): [18:26:51] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:27:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:28:51] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:31:30] Free Memory on damiana is OK: OK - 7.3% (609424 kB) free. [18:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:01] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.03, 26.12, 17.10 [18:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.71, 37.55, 37.42 [18:53:41] / on ptolemy is WARNING: DISK WARNING - free space: / 2435 MB (12% inode=91%): [18:53:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:54:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 421622 MB (7% inode=38%): [19:01:31] Free Memory on damiana is WARNING: WARNING - 6.9% (580864 kB) free! [19:05:31] Free Memory on damiana is OK: OK - 7.1% (592568 kB) free. [19:10:01] Load avg. on ortelius is WARNING: WARNING - load average: 10.74, 14.41, 19.60 [19:17:31] Free Memory on damiana is WARNING: WARNING - 6.8% (570968 kB) free! [19:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124667 MB (20% inode=99%): [19:26:31] Free Memory on damiana is OK: OK - 7.2% (599620 kB) free. [19:30:01] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 21.18, 21.54, 20.06 [19:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:38:31] Free Memory on damiana is WARNING: WARNING - 6.9% (582088 kB) free! [19:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.65, 33.51, 34.32 [19:53:42] / on ptolemy is WARNING: DISK WARNING - free space: / 2242 MB (11% inode=91%): [19:53:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:54:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 421452 MB (7% inode=38%): [19:57:01] Load avg. on ortelius is WARNING: WARNING - load average: 8.30, 16.03, 19.36 [20:08:41] / on ptolemy is CRITICAL: DISK CRITICAL - free space: / 2194 MB (10% inode=91%): [20:14:01] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 32.93, 24.57, 19.92 [20:18:42] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:19:01] Load avg. on ortelius is WARNING: WARNING - load average: 13.85, 20.32, 19.60 [20:19:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124499 MB (20% inode=99%): [20:25:39] DaBPunkt: hello. Any word yet from WMF on replacing our corrupt copies of enwiki? [20:26:50] russblau: A graveyard at midnight is louder ;). I will send a reminder tomorrow [20:28:27] may they are faster with setting up replication on labs than giving us a new dump … [20:29:23] *maybe [20:29:45] but makes sense anyway [20:31:42] DaBPunkt: great, by tomorrow they should all be recovered from their New Year's hangovers :-) [20:32:01] Load avg. on ortelius is OK: OK - load average: 8.23, 10.91, 14.72 [20:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:53:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:53:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.98, 37.32, 36.14 [20:53:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:54:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 421270 MB (7% inode=38%): [21:08:41] / on ptolemy is CRITICAL: DISK CRITICAL - free space: / 2004 MB (10% inode=91%): [21:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124280 MB (20% inode=99%): [21:20:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:37:31] Free Memory on damiana is WARNING: WARNING - 6.1% (515184 kB) free! [21:43:31] Free Memory on damiana is OK: OK - 7.5% (630456 kB) free. [21:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:53:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.64, 35.23, 35.81 [21:53:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:54:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 421100 MB (7% inode=38%): [21:59:41] / on ptolemy is OK: DISK OK - free space: / 10647 MB (53% inode=91%): [22:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124090 MB (20% inode=99%): [22:20:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:27:31] /sql on cassia is WARNING: DISK WARNING - free space: /sql 102984 MB (8% inode=99%): [22:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:53:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:53:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.79, 36.39, 36.26 [22:53:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:54:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 420944 MB (7% inode=38%): [23:09:47] nacht ts [23:19:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123929 MB (20% inode=99%): [23:20:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:48:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:53:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:53:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:53:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.55, 36.59, 36.39 [23:53:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:54:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:54:11] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 420096 MB (7% inode=38%):