[00:00:59] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:11:09] / on damiana is WARNING: DISK WARNING - free space: / 13663 MB (19% inode=95%): [00:17:29] MySQL slave on rosemary is OK: Uptime: 30930 Threads: 17 Questions: 30438342 Slow queries: 4926 Opens: 3481 Flush tables: 1 Open tables: 657 Queries per second avg: 984.104 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [00:18:19] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1695.000000 [00:19:18] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [00:36:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:38:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 636476.000000 [00:39:09] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:40:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72960 MB (11% inode=99%): [00:42:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:44:59] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:49:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:49:59] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:07:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:11:09] / on damiana is WARNING: DISK WARNING - free space: / 11956 MB (16% inode=95%): [01:15:09] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.26, 23.88, 13.62 [01:16:09] Load avg. on ortelius is WARNING: WARNING - load average: 25.89, 23.84, 14.27 [01:19:18] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [01:20:09] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 27.79, 25.41, 17.11 [01:36:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:37:49] nacht ts [01:38:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 621175.000000 [01:38:15] Night, DaB. [01:39:10] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [01:40:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71541 MB (11% inode=99%): [01:49:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:07:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:11:09] / on damiana is WARNING: DISK WARNING - free space: / 10145 MB (14% inode=95%): [02:12:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:18:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:19:18] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [02:20:09] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 24.55, 25.19, 25.22 [02:25:20] / on ortelius is WARNING: DISK WARNING - free space: / 6190 MB (20% inode=92%): [02:26:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:36:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:38:09] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:38:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 605294.000000 [02:38:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:39:09] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:40:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72790 MB (11% inode=99%): [02:44:10] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:09] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:47:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:49:10] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:50:09] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:50:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:53:10] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:59:09] Load avg. on ortelius is WARNING: WARNING - load average: 9.99, 12.09, 19.23 [03:04:10] Load avg. on ortelius is OK: OK - load average: 3.16, 6.67, 14.82 [03:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:11:10] / on damiana is WARNING: DISK WARNING - free space: / 8896 MB (12% inode=95%): [03:18:10] / on damiana is CRITICAL: DISK CRITICAL - free space: / 7768 MB (10% inode=95%): [03:19:19] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [03:26:19] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 3166 MB (10% inode=92%): [03:27:19] / on ortelius is OK: DISK OK - free space: / 11497 MB (38% inode=92%): [03:36:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:38:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 578853.000000 [03:39:09] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:39:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:40:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72712 MB (11% inode=99%): [03:49:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:02:09] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 72793 MB (7% inode=99%): [04:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:18:09] / on damiana is CRITICAL: DISK CRITICAL - free space: / 7346 MB (10% inode=95%): [04:19:19] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [04:36:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:38:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 561104.000000 [04:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:40:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72642 MB (11% inode=99%): [04:49:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:03:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:06:19] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:06:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:08:19] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:18:20] / on damiana is CRITICAL: DISK CRITICAL - free space: / 5989 MB (8% inode=95%): [05:19:19] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [05:36:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:38:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 550562.000000 [05:39:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:39:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:40:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72587 MB (11% inode=99%): [05:49:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:13:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:18:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 4443 MB (6% inode=95%): [06:19:19] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [06:21:19] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:36:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:36:50] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:38:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 547707.000000 [06:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:40:20] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72540 MB (11% inode=99%): [06:40:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:49:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:53:19] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:53:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:57:19] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:57:20] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:57:20] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:57:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:58:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:59:19] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:01:19] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:18:20] / on damiana is CRITICAL: DISK CRITICAL - free space: / 3035 MB (4% inode=95%): [07:19:19] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [07:38:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 528096.000000 [07:39:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:39:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:40:20] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72469 MB (11% inode=99%): [07:49:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:07:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:18:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:19:19] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [08:19:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 1812 MB (2% inode=95%): [08:24:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:24:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:38:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 511375.000000 [08:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:41:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72371 MB (11% inode=99%): [08:49:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:59:19] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:59:20] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:59:20] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:59:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:59:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:01:20] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:01:20] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:11:19] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:19:20] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [09:19:20] / on damiana is CRITICAL: DISK CRITICAL - free space: / 244 MB (0% inode=95%): [09:38:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 497386.000000 [09:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:41:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72265 MB (11% inode=99%): [09:49:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:53:19] /sql on cassia is WARNING: DISK WARNING - free space: /sql 100762 MB (8% inode=99%): [10:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:19:20] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [10:19:20] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [10:38:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 487579.000000 [10:39:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:39:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:41:20] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72138 MB (11% inode=99%): [10:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:01:14] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 34.25, 24.64, 14.16 [11:05:19] Free Memory on damiana is CRITICAL: CRITICAL - 1.6% (134576 kB) free! [11:06:09] Load avg. on ortelius is WARNING: WARNING - load average: 17.58, 24.62, 17.45 [11:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:11:09] Load avg. on ortelius is OK: OK - load average: 8.18, 14.39, 14.92 [11:16:20] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 85786 MB (1% inode=50%): [11:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:19:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7493 [11:19:20] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [11:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:26:19] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3166 [11:29:19] MySQL slave on z-dat-s6-a is OK: Uptime: 255995 Threads: 8 Questions: 61293809 Slow queries: 12120 Opens: 379899 Flush tables: 1 Open tables: 2930 Queries per second avg: 239.433 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1313 [11:38:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 484597.000000 [11:39:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:42:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71976 MB (11% inode=99%): [11:45:20] /tmp on damiana is WARNING: DISK WARNING - free space: /tmp 1118 MB (11% inode=98%): [11:46:19] /tmp on damiana is CRITICAL: DISK CRITICAL - free space: /tmp 943 MB (9% inode=98%): [11:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:53:09] Load avg. on ortelius is WARNING: WARNING - load average: 18.00, 15.72, 12.20 [11:55:09] Load avg. on ortelius is OK: OK - load average: 11.20, 14.09, 12.03 [12:05:20] Free Memory on damiana is CRITICAL: CRITICAL - 2.6% (214568 kB) free! [12:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:16:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 85786 MB (1% inode=50%): [12:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:20:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [12:22:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:27:20] / on ortelius is WARNING: DISK WARNING - free space: / 6190 MB (20% inode=92%): [12:38:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 484252.000000 [12:39:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:42:20] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71826 MB (11% inode=99%): [12:46:19] /tmp on damiana is CRITICAL: DISK CRITICAL - free space: /tmp 578 MB (6% inode=98%): [12:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:05:19] Free Memory on damiana is CRITICAL: CRITICAL - 2.5% (206320 kB) free! [13:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:16:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 85786 MB (1% inode=50%): [13:18:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:20:32] / on damiana is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): [13:21:19] Free Memory on damiana is WARNING: WARNING - 5.5% (463972 kB) free! [13:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:23:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:27:19] / on ortelius is WARNING: DISK WARNING - free space: / 5696 MB (19% inode=92%): [13:28:20] Free Memory on damiana is CRITICAL: CRITICAL - 3.7% (309576 kB) free! [13:32:24] [[Special:Log/newusers]] create 10 * Nancyyo * (New user account) [13:34:53] [[User talk:Nancyyo]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7810&rcid=21545 * Nancyyo * (+3215) (Created page with "Some Effect With Alcohol Abuse On Your Body. Alcohol abuse might cause injury to many areas of the body. Beyond the heart, abusive drinking can impact smaller sized organs suc...") [13:38:19] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 473724.000000 [13:38:20] Free Memory on damiana is WARNING: WARNING - 5.1% (423224 kB) free! [13:39:19] / on damiana is OK: DISK OK - free space: / 20406 MB (28% inode=95%): [13:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:39:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:42:20] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71663 MB (11% inode=99%): [13:46:19] /tmp on damiana is CRITICAL: DISK CRITICAL - free space: /tmp 594 MB (6% inode=98%): [13:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:58:19] Free Memory on damiana is CRITICAL: CRITICAL - 2.5% (210232 kB) free! [14:03:08] @replag s2 [14:03:13] saper: s1-rr-a-wd: 58s [-0.00 s/s]; s1-user-wd: 58s [-0.00 s/s]; s2-rr: 4m 29s [-0.03 s/s]; s2-user: 4m 29s [-0.03 s/s]; s2-user-c: error; s2-user-wd: 12h 35m 0s [+0.44 s/s]; s3-user: 16s [-0.00 s/s]; s4-user-wd: 1m 2s [-0.00 s/s] [14:03:14] saper: s5-rr-a: 35s [-]; s5-user: 35s [-]; s5-user-c: 5d 9h 40m 57s [-3.57 s/s]; s5-user-wd: 1m 54s [+0.00 s/s]; s6-user-wd: 1m 2s [-0.00 s/s]; s7-user-wd: 1m 2s [-0.00 s/s] [14:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:16:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 84837 MB (1% inode=50%): [14:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:27:19] / on ortelius is WARNING: DISK WARNING - free space: / 5250 MB (17% inode=92%): [14:37:07] hello all [14:37:12] @replag [14:37:13] DaBPunkt: s1-rr-a-wd: 1m 23s [+0.01 s/s]; s1-user: 1m 5s [-0.08 s/s]; s1-user-wd: 1m 23s [+0.01 s/s]; s2-rr: 26m 23s [+0.64 s/s]; s2-user: 26m 23s [+0.64 s/s]; s2-user-c: error; s2-user-wd: 13h 9m 3s [+1.00 s/s]; s3-user: 16s [-] [14:37:13] DaBPunkt: s4-user-wd: 1m 24s [+0.01 s/s]; s5-user-c: 5d 7h 22m 44s [-4.07 s/s]; s5-user-wd: 1m 23s [-0.02 s/s]; s6-user-wd: 1m 24s [+0.01 s/s]; s7-user-wd: 1m 24s [+0.01 s/s] [14:38:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 458516.000000 [14:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:39:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:42:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71479 MB (11% inode=99%): [14:45:19] / on damiana is WARNING: DISK WARNING - free space: / 14416 MB (20% inode=95%): [14:46:20] /tmp on damiana is CRITICAL: DISK CRITICAL - free space: /tmp 613 MB (6% inode=98%): [14:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:58:06] DaB. * Re: [Toolserver-announce] [Toolserver-l] Postmortem: Partial Toolserver-outage [14:58:19] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (415144 kB) free! [15:03:29] Free Memory on damiana is WARNING: WARNING - 5.5% (458108 kB) free! [15:04:29] /tmp on damiana is OK: DISK OK - free space: /tmp 9521 MB (99% inode=98%): [15:04:30] Free Memory on damiana is OK: OK - 35.7% (2995212 kB) free. [15:05:17] [[Special:Log/newusers]] create 10 * Nancyyon * (New user account) [15:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:07:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:16:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 84538 MB (1% inode=50%): [15:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:27:20] / on ortelius is WARNING: DISK WARNING - free space: / 5086 MB (16% inode=92%): [15:34:33] dschwen: did you change anything at your map-tool? Out access-log-size is increasing very fast [15:34:40] Out → Our [15:38:28] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 445619.000000 [15:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:39:29] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:42:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71305 MB (11% inode=99%): [15:44:23] hmm [15:44:37] this may be related to the absence of the cache dir [15:44:42] let me have a look [15:45:29] / on damiana is WARNING: DISK WARNING - free space: / 11479 MB (15% inode=95%): [15:46:13] ok, DaBPunkt, once the outdated JS is out of the caches it should calm down now [15:46:30] it was due to my on-demand tile rendering algorithm [15:46:50] the cache dir is absent (/mnt/user-store) so rendering fails [15:47:05] but the script keeps polling to see if the tile has finished rendering [15:47:14] disabled that function [15:48:42] oh, just saw your email [15:48:45] user-store is back [15:48:49] hm [15:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:50:53] s2 will be away for a moment [16:02:29] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 72063 MB (7% inode=99%): [16:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:16:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 86191 MB (1% inode=50%): [16:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:27:19] / on ortelius is WARNING: DISK WARNING - free space: / 4985 MB (16% inode=92%): [16:35:29] / on damiana is OK: DISK OK - free space: / 26595 MB (36% inode=95%): [16:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 441939.000000 [16:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71103 MB (11% inode=99%): [16:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:53:25] DaBPunkt, is the access log still growing fast? [17:06:48] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:16:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 96353 MB (1% inode=51%): [17:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:20:08] man, deleting a bazillion files sure takes a long time [17:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 4816 MB (16% inode=92%): [17:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 439496.000000 [17:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70903 MB (11% inode=99%): [17:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:16:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 106054 MB (1% inode=51%): [18:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 4678 MB (15% inode=92%): [18:36:06] Carl (CBM) * Re: [Toolserver-l] Cron on submit [18:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 433699.000000 [18:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70700 MB (11% inode=99%): [18:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:51:01] DaBPunkt, I'm currently deleting an - as of today - obslete tree of cached tiles on /mnt/user-store [18:51:26] this rm -rf is going to take several hours from my current projections [18:51:43] is this ok, or am I causing unacceptable filesystem load? [18:55:10] also: it seems that not all users have heard of the SGE-only rule... [19:02:02] yeah, well, I'm going to lunch soon, I'll just assume its ok unless i hear otherwise. It will free a considerable amount of space on user-store after all. An I guess the admins know how to kill a process ;-). [19:06:48] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:07:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:16:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 115827 MB (2% inode=51%): [19:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:22:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 4542 MB (15% inode=92%): [19:27:40] @replag [19:27:40] DaBPunkt: s1-rr-a-wd: 2m 56s [+0.01 s/s]; s1-user-wd: 2m 56s [+0.01 s/s]; s2-rr: 12s [-0.09 s/s]; s2-user: 12s [-0.09 s/s]; s2-user-c: error; s2-user-wd: 2m 56s [-2.71 s/s]; s4-user-wd: 2m 56s [+0.01 s/s]; s5-user-c: 4d 22h 8m 2s [-1.91 s/s] [19:27:41] DaBPunkt: s5-user-wd: 2m 56s [+0.01 s/s]; s6-user-wd: 2m 56s [+0.01 s/s]; s7-user-wd: 2m 56s [+0.01 s/s] [19:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 423047.000000 [19:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70423 MB (11% inode=99%): [19:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:16:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 127487 MB (2% inode=52%): [20:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 4352 MB (14% inode=92%): [20:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 404515.000000 [20:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:42:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70185 MB (11% inode=99%): [20:49:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:06:06] DaBPunkt, I noticed /mnt/user-store seems to be mounted via NFS from rosemary. Anychance somone could log on to rosemary to delete sime of my files locally. Deleting via NFS seems to take forever. [21:06:16] sure [21:06:41] I cannot login myself, can I? [21:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:06:51] no [21:07:10] the directory to be deleted is /mnt/user-store/wikiminiatlas/jsontile.old/ [21:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:07:52] there are a few GB to be freed there [21:09:25] running [21:09:31] thanks [21:09:35] I modified my code today to store gzipped cache files [21:09:50] should reduce my needed space by a factor of about 5 [21:10:11] how do these titles differ from those of the OSM-people? [21:11:11] they contain JSON encoded data which is rendered in the browser [21:11:24] this allows me to display 3D buildings [21:11:28] (using webGL) [21:11:35] looks pretty cool [21:12:56] example? [21:13:01] Also this data allows me to zoom further into the map without additional network traffic [21:13:05] oh, one sec [21:13:20] well, go to the [[Empire State Building]] article [21:13:24] german or english [21:13:34] open the WMA and zoom in [21:15:24] ah ok [21:15:30] you should see 3d buildings unless you ar using an old Internet Explorer (which I would find hard to believe) [21:15:32] ;-) [21:15:56] in big cities there is a good amount of 3d building data on OSM [21:16:05] some are incredibly detailed [21:16:18] some, of course, are just boxes [21:16:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 138790 MB (2% inode=52%): [21:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:20:08] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [21:22:09] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [21:22:42] looks like deletion is still slow. it's a lot of small files. [21:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:23:03] dschwen: there are other users too [21:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:24:24] ? [21:24:36] are you reminding me to be considerate? [21:25:02] or are you pointing out that other users access to user-store could slow things down? [21:25:15] dschwen: that's the reason deletion of small files take a while. Because other users use the array at the same time [21:25:28] ok [21:25:34] well thre is no rush I guess [21:25:50] no, 138gb is a safe value [21:26:08] it was at 96GB when I started deleting... [21:26:19] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [21:26:36] @replag [21:26:36] DaBPunkt: s1-rr-a: 4m 26s [-]; s1-rr-a-wd: 1m 44s [-0.01 s/s]; s1-user: 4m 26s [+0.01 s/s]; s1-user-wd: 1m 46s [-0.01 s/s]; s2-rr: 47m 39s [+0.40 s/s]; s2-user: 47m 39s [+0.40 s/s]; s2-user-c: error; s2-user-wd: 3m 6s [+0.00 s/s] [21:26:37] DaBPunkt: s3-user: 16s [-]; s4-rr-a: 13m 18s [-]; s4-user: 13m 18s [-]; s4-user-wd: 1m 44s [-0.01 s/s]; s5-rr-a: 10m 32s [+0.02 s/s]; s5-user: 10m 32s [+0.02 s/s]; s5-user-c: 4d 10h 34m 30s [-5.83 s/s]; s5-user-wd: 1m 46s [-0.01 s/s] [21:26:38] DaBPunkt: s6-user: 7m 2s [-0.01 s/s]; s6-user-wd: 1m 46s [-0.01 s/s]; s7-user: 5m 12s [-]; s7-user-wd: 1m 46s [-0.01 s/s] [21:26:41] I may have to come up with a strategy of pruning cachefiles that haven't been accessed for a few days [21:27:02] the fs does have 'atime', right? [21:27:10] yes [21:27:22] then a simple find once a week or so should work [21:27:28] / on ortelius is WARNING: DISK WARNING - free space: / 4221 MB (14% inode=92%): [21:27:40] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [21:27:42] can I request an SGQ job to be run on rosemary? [21:27:48] SGE [21:27:48] no [21:28:04] does every SGE node have /mnt/user-store mounted? [21:28:10] yes [21:28:18] or should at least [21:28:22] ok [21:28:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:29:09] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [21:29:13] by the way, I looke at the SGE tutorial, and one thing is still not clear to me [21:29:45] yes? [21:29:46] when porting my bot launch scripts to use SGE I need to request SQL resources [21:30:11] I still do not know the mapping between all the various sql host aliases to SGE resources [21:31:15] hold on [21:31:21] don't answer [21:31:49] doing a 'host sqlhostalias' should help [21:32:32] hap-s4-user.esi.toolserver.org would be sql-s4-user I guess [21:32:51] ok, nevermind then [21:32:52] that'a a possiblity. Another way is to use the toolserver.wiki-table. [21:32:59] It has a server-field [21:34:38] oh, does -rr- stand for round robin' ? [21:34:44] yes [21:34:55] it all starts to make sense ;-) [21:36:11] there is btw a fs-user-store-resource [21:38:28] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 379174.000000 [21:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69979 MB (11% inode=99%): [21:44:37] oh ok, thanks [21:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:50:19] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2214.000000 [21:52:09] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2162.000000 [21:53:39] /sql on cassia is WARNING: DISK WARNING - free space: /sql 102504 MB (8% inode=99%): [21:55:09] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1940 [21:55:09] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2427 [21:56:09] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1949 [21:56:09] MySQL slave on cassia is OK: Uptime: 5187730 Threads: 10 Questions: 13245685528 Slow queries: 802986 Opens: 18468511 Flush tables: 3 Open tables: 14238 Queries per second avg: 2553.271 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1428 [21:56:09] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1431.000000 [21:56:19] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1783.000000 [21:56:39] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2032 [21:57:09] MySQL slave on daphne is OK: Uptime: 3888121 Threads: 25 Questions: 873185823 Slow queries: 45764 Opens: 174322 Flush tables: 1 Open tables: 1324 Queries per second avg: 224.577 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1350 [21:58:19] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1804.000000 [21:58:29] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2030.000000 [21:59:09] MySQL slave on thyme is OK: Uptime: 5185775 Threads: 11 Questions: 2781754693 Slow queries: 345428 Opens: 117089 Flush tables: 1 Open tables: 681 Queries per second avg: 536.420 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1768 [21:59:18] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1763.000000 [22:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:07:11] @replag [22:07:12] DaBPunkt: s1-rr-a: 26m 42s [+0.55 s/s]; s1-rr-a-wd: 1m 38s [-0.00 s/s]; s1-user: 33m 17s [+0.71 s/s]; s1-user-wd: 1m 38s [-0.00 s/s]; s2-rr: 1h 1m 36s [+0.34 s/s]; s2-user: 1h 1m 36s [+0.34 s/s]; s2-user-c: error; s2-user-wd: 1m 38s [-0.04 s/s] [22:07:12] DaBPunkt: s3-user: 11s [-0.00 s/s]; s4-user-wd: 1m 41s [-0.00 s/s]; s5-user-c: 4d 7h 4m 30s [-5.17 s/s]; s5-user-wd: 1m 38s [-0.00 s/s]; s6-user: 47m 37s [+1.00 s/s]; s6-user-wd: 1m 37s [-0.00 s/s]; s7-user: 45m 47s [+1.00 s/s]; s7-user-wd: 1m 37s [-0.00 s/s] [22:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:10:19] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2896 [22:10:23] @replag [22:10:23] DaBPunkt: s1-rr-a: 24m 45s [-0.61 s/s]; s1-rr-a-wd: 1m 49s [+0.06 s/s]; s1-user: 32m 52s [-0.13 s/s]; s1-user-wd: 1m 46s [+0.04 s/s]; s2-rr: 1h 42s [-0.28 s/s]; s2-user: 1h 42s [-0.28 s/s]; s2-user-c: error; s2-user-wd: 1m 50s [+0.06 s/s] [22:10:24] DaBPunkt: s4-user-wd: 1m 47s [+0.03 s/s]; s5-user-c: 4d 6h 54m 17s [-3.20 s/s]; s5-user-wd: 1m 50s [+0.06 s/s]; s6-user: 48m 37s [+0.31 s/s]; s6-user-wd: 1m 47s [+0.05 s/s]; s7-user: 48m 46s [+0.93 s/s]; s7-user-wd: 1m 50s [+0.07 s/s] [22:10:40] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2919 [22:16:19] MySQL slave on z-dat-s6-a is OK: Uptime: 294817 Threads: 6 Questions: 68977084 Slow queries: 12983 Opens: 381266 Flush tables: 1 Open tables: 2947 Queries per second avg: 233.965 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1468 [22:16:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 152201 MB (2% inode=52%): [22:17:32] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1793.000000 [22:17:42] MySQL slave on rosemary is OK: Uptime: 110142 Threads: 9 Questions: 72962721 Slow queries: 20029 Opens: 8881 Flush tables: 1 Open tables: 733 Queries per second avg: 662.442 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1787 [22:17:42] MySQL slave on z-dat-s7-a is OK: Uptime: 294897 Threads: 3 Questions: 105850092 Slow queries: 7151 Opens: 491766 Flush tables: 1 Open tables: 4354 Queries per second avg: 358.939 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1744 [22:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:19:16] @replag [22:19:16] DaBPunkt: s1-rr-a: 7m 12s [-1.98 s/s]; s1-rr-a-wd: 1m 50s [+0.00 s/s]; s1-user: 27m 37s [-0.59 s/s]; s1-user-wd: 1m 49s [+0.01 s/s]; s2-rr: 1h 9m 36s [+1.00 s/s]; s2-user: 1h 9m 36s [+1.00 s/s]; s2-user-c: error; s2-user-wd: 1m 53s [+0.01 s/s] [22:19:17] DaBPunkt: s4-user-wd: 1m 49s [+0.00 s/s]; s5-user-c: 4d 6h 24m 30s [-3.35 s/s]; s5-user-wd: 1m 50s [-]; s6-user: 8m 27s [-4.52 s/s]; s6-user-wd: 1m 50s [+0.01 s/s]; s7-user: 21m 23s [-3.08 s/s]; s7-user-wd: 1m 50s [-] [22:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 4101 MB (13% inode=92%): [22:38:28] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367499.000000 [22:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:40:39] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:42:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69837 MB (11% inode=99%): [22:45:44] @replag [22:45:44] DaBPunkt: s1-rr-a-wd: 1m 54s [+0.00 s/s]; s1-user-wd: 2m 0s [+0.01 s/s]; s2-rr: 1h 4m 38s [-0.19 s/s]; s2-user: 1h 4m 38s [-0.19 s/s]; s2-user-c: error; s2-user-wd: 2m 0s [+0.00 s/s]; s4-user-wd: 2m 0s [+0.01 s/s]; s5-user-c: 4d 5h 59m 56s [-0.93 s/s] [22:45:45] DaBPunkt: s5-user-wd: 2m 0s [+0.01 s/s]; s6-user-wd: 2m 0s [+0.01 s/s]; s7-user-wd: 1m 54s [+0.00 s/s] [22:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:00:49] nacht ts [23:06:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:16:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 166310 MB (3% inode=53%): [23:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 4004 MB (13% inode=92%): [23:38:28] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 359674.000000 [23:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:40:38] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:42:28] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69699 MB (11% inode=99%): [23:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default