[00:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [00:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:18:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95298 MB (15% inode=99%): [00:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 304674 MB (5% inode=30%): [00:49:52] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3581 [00:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:52:52] MySQL slave on z-dat-s7-a is OK: Uptime: 33289 Threads: 9 Questions: 27406876 Slow queries: 394 Opens: 4414 Flush tables: 1 Open tables: 1205 Queries per second avg: 823.301 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1667 [00:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.85, 37.09, 36.52 [00:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:03:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [01:04:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:07:52] Free Memory on damiana is WARNING: WARNING - 5.8% (484604 kB) free! [01:14:52] Free Memory on damiana is OK: OK - 7.1% (591140 kB) free. [01:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95142 MB (15% inode=99%): [01:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 303119 MB (5% inode=30%): [01:41:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:43:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:52:07] Susan: when will the database info be back? (CA information seems to be missing) [01:52:29] Jasper_Deng: I think that's s7. There was toolserver-l mail about it. [01:52:43] http://status.toolserver.org/ is updated. \o/ [01:53:05] Somewhat, at least. s1 still looks fine. [01:53:10] Even though I believe it's corrupt. [01:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:54:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.46, 31.45, 32.10 [01:54:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:55:52] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [02:05:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93219 MB (15% inode=99%): [02:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 302497 MB (5% inode=30%): [02:27:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:29:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:30:02] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:31:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:32:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.82, 33.30, 32.86 [02:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 88245 MB (9% inode=99%): [02:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [03:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:13:38] anyone with a TS account around? [03:13:47] I'd like to try to get the contributions of a user [03:14:25] specifically, the enwiki contributions of User:%C9%B1(where that is the charactor code) [03:16:27] Susan, it is sort of an interesting project, if you are interested :) [03:16:52] there is some background here - http://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)/Archive_105#Link_in_history_says_user_does_not_exist [03:18:13] Fun. [03:18:25] Is there an associated bug? [03:18:49] it looks like no one ever filed one [03:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:19:24] you can read the link, but basically the consensus was that this is basically like having a User:a [03:19:31] so the bug is that the account exists [03:20:42] Right. [03:20:53] I replied in -tech. [03:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94831 MB (15% inode=99%): [03:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 303845 MB (5% inode=30%): [03:31:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:32:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.09, 31.43, 31.46 [03:50:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:08:22] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:08:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:15:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94761 MB (15% inode=99%): [04:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 303121 MB (5% inode=30%): [04:27:22] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:32:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.07, 30.89, 31.18 [04:34:02] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:34:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:35:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.55, 29.79, 30.72 [04:35:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [05:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:13:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:14:02] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.22, 27.52, 27.64 [05:16:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94677 MB (15% inode=99%): [05:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 303685 MB (5% inode=30%): [05:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:30:21] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.11, 28.03, 27.21 [06:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94575 MB (15% inode=99%): [06:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 302981 MB (5% inode=30%): [06:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.67, 31.66, 31.98 [07:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94473 MB (15% inode=99%): [07:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 302170 MB (5% inode=30%): [07:27:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:41:52] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420044 kB) free! [07:45:52] Free Memory on damiana is WARNING: WARNING - 6.3% (525976 kB) free! [07:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:50:52] Free Memory on damiana is OK: OK - 7.8% (653364 kB) free. [07:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [08:05:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.56, 32.75, 32.27 [08:18:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94341 MB (15% inode=99%): [08:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 300705 MB (5% inode=30%): [08:27:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [09:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.31, 31.16, 31.42 [09:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94173 MB (15% inode=99%): [09:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 298416 MB (5% inode=30%): [09:27:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:54:52] Free Memory on damiana is WARNING: WARNING - 7.0% (585420 kB) free! [09:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:07:21] what's the best IRC channel to ask questions about the XML dump files? [10:07:52] Free Memory on damiana is OK: OK - 7.4% (621788 kB) free. [10:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.54, 29.33, 29.83 [10:18:02] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2023.000000 [10:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93991 MB (15% inode=99%): [10:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 300190 MB (5% inode=30%): [10:27:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 89182 MB (7% inode=99%): [10:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:49:53] Free Memory on damiana is WARNING: WARNING - 6.2% (520384 kB) free! [10:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:53:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (380864 kB) free! [10:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:59:52] Free Memory on damiana is WARNING: WARNING - 5.2% (434372 kB) free! [11:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.75, 29.55, 29.67 [11:18:02] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2156.000000 [11:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93795 MB (15% inode=99%): [11:26:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 297076 MB (5% inode=30%): [11:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:47:52] Free Memory on damiana is WARNING: WARNING - 7.0% (584508 kB) free! [11:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:52:02] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1791.000000 [11:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:57:52] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:57:52] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [12:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [12:05:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:05:53] Free Memory on damiana is CRITICAL: CRITICAL - 3.5% (292168 kB) free! [12:13:52] MySQL slave on z-dat-s3-a is OK: Uptime: 1334 Threads: 10 Questions: 2352 Slow queries: 0 Opens: 386 Flush tables: 1 Open tables: 375 Queries per second avg: 1.763 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1358 [12:13:52] MySQL on z-dat-s3-a is OK: Uptime: 1334 Threads: 9 Questions: 2354 Slow queries: 0 Opens: 386 Flush tables: 1 Open tables: 375 Queries per second avg: 1.764 [12:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.00, 30.52, 30.64 [12:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93614 MB (15% inode=99%): [12:24:05] has anyone else had problems with cron on willow in the past couple days? [12:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 296807 MB (5% inode=30%): [12:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:28:02] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3491 MB (18% inode=99%): [12:34:52] Free Memory on damiana is WARNING: WARNING - 5.4% (450616 kB) free! [12:38:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (404960 kB) free! [12:39:52] Free Memory on damiana is WARNING: WARNING - 5.1% (431336 kB) free! [12:46:04] Carl (CBM) * [Toolserver-l] Cron on willow [12:50:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:53:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:00:12] that doesn't look very good... [13:02:52] Free Memory on damiana is WARNING: WARNING - 5.6% (467632 kB) free! [13:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [13:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:14:04] Platonides * Re: [Toolserver-l] Cron on willow [13:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.08, 29.19, 29.86 [13:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93333 MB (15% inode=99%): [13:21:39] hi, replag is not updated since a few days : http://toolserver.org/~bryan/stats/replag/ [13:22:52] Free Memory on damiana is OK: OK - 7.2% (603776 kB) free. [13:25:04] Carl (CBM) * Re: [Toolserver-l] Cron on willow [13:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 295594 MB (5% inode=30%): [13:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:28:02] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3433 MB (18% inode=99%): [13:29:52] Free Memory on damiana is WARNING: WARNING - 6.6% (554552 kB) free! [13:32:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (386848 kB) free! [13:36:04] Platonides * Re: [Toolserver-l] Cron on willow [13:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:57:04] DaB. * Re: [Toolserver-announce] [Toolserver-l] Downtime of s7 starting tomorrow [14:02:04] Carl (CBM) * Re: [Toolserver-l] Cron on willow [14:02:40] uh oh [14:02:57] yeah, i was about to menbtion that [14:03:42] there are a lot of reza scripts hogging all the CPU, apparently this caused crons to not run correctly as well [14:04:20] okay, it's talking about another cron, but.. hey. still, there's a problem [14:04:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:13:04] Wolfgang ten Weges * [Toolserver-l] Crons overflow/server overload on Willow (other) [14:13:31] there :) [14:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.50, 32.00, 30.78 [14:18:35] phe: /quit [14:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:20:26] @replag [14:20:27] DarkoNeko: s1-rr-a-wd: 16s [-0.00 s/s]; s1-user-wd: 16s [-0.00 s/s]; s2-user-wd: 14s [-0.00 s/s]; s3-user: 12s [-0.00 s/s]; s4-user-wd: 14s [-0.00 s/s]; s5-user-wd: 14s [-0.00 s/s]; s6-user-wd: 16s [-0.00 s/s]; s7-user-wd: 14s [-0.00 s/s] [14:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93113 MB (15% inode=99%): [14:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 293502 MB (5% inode=30%): [14:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:28:02] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3010 MB (16% inode=99%): [14:28:04] Platonides * Re: [Toolserver-l] Cron on willow [14:31:50] urg, feels weird being called by my real name on the ML [14:33:04] Hydriz Wikipedia * Re: [Toolserver-l] Downtime of s7 starting tomorrow [14:33:27] wooooooooooooooooooooolfgaaaaaaang [14:33:29] ;-) [14:33:48] you just recently started using your real name on MLs? [14:33:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (412800 kB) free! [14:34:52] Free Memory on damiana is WARNING: WARNING - 5.6% (467432 kB) free! [14:40:52] Free Memory on damiana is OK: OK - 7.2% (602916 kB) free. [14:42:00] no, it's been shown next to my mail address. but usually people just use the nick [14:42:44] CRON wise, it's the not the first time i see it borked by someone running way too many script ; i wish people were more careful... [14:44:16] yeah. it should really only be used for short-running jobs [14:44:20] if at all [14:44:51] but the entire 'we have cronie, crontabs, SGE, ...'-stuff is not really simple anyway [14:45:52] urg, use "shuaib" has 94 scripts or things opened, too [14:47:14] oh come on, most are "welcomeur.py" ? dont keep that runing, use -break ffs [14:47:52] wtf is that anyway? some arabic adaptation of welcome.py? [14:48:26] it sounds like a french adverb, actually. Bu I don't know that shuaib person [14:48:39] welcomeur : "one that welcomes" [14:48:41] ah, 'the welcomer' [14:48:46] yes, that [14:48:58] it's ugly frengrish but that's a possibility [14:49:15] the only reference I can fidn is tahirbot [14:49:29] on urwiki, actually, not arwiki [14:49:39] what language is ur ? [14:49:45] urdu [14:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:50:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 87489 MB (8% inode=99%): [14:51:00] urg, "grep" seems to limit to the first 80 chars of each line. any idea how to get around that ? [14:51:23] or maybe pipe ( | ) is directly reesponsible. grumbl [14:53:04] Wolfgang ten Weges * Re: [Toolserver-l] Crons overflow/server overload on Willow (other) [14:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:05:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:07:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (368708 kB) free! [15:08:52] Free Memory on damiana is WARNING: WARNING - 5.1% (429132 kB) free! [15:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.95, 31.14, 30.82 [15:15:52] Free Memory on damiana is OK: OK - 7.1% (596232 kB) free. [15:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92904 MB (15% inode=99%): [15:24:52] Free Memory on damiana is WARNING: WARNING - 5.8% (488176 kB) free! [15:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292298 MB (5% inode=29%): [15:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:28:02] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3481 MB (18% inode=99%): [15:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:51:08] phil.el * Re: [Toolserver-l] Cron on willow [15:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:04:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:05:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.82, 33.89, 32.93 [16:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92687 MB (15% inode=99%): [16:26:07] DaB. * Re: [Toolserver-l] Downtime of s7 starting tomorrow [16:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291005 MB (5% inode=29%): [16:27:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:28:02] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3440 MB (18% inode=99%): [16:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:53:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:04:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (1 error, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5: [17:05:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:10:13] Danny B. * Re: [Toolserver-l] [17:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.64, 31.76, 31.47 [17:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92431 MB (15% inode=99%): [17:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290240 MB (5% inode=29%): [17:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:28:02] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3204 MB (17% inode=99%): [17:50:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:15:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.41, 32.35, 31.94 [18:15:08] Wolfgang ten Weges * Re: [Toolserver-l] Crons overflow/server overload on Willow (other) [18:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92166 MB (15% inode=99%): [18:26:07] John * Re: [Toolserver-l] Crons overflow/server overload on Willow (other) [18:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292314 MB (5% inode=29%): [18:27:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:28:02] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2946 MB (16% inode=99%): [18:45:08] Platonides * Re: [Toolserver-l] Crons overflow/server overload on Willow (other) [18:51:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:54:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:55:01] /tmp on ortelius is OK: DISK OK - free space: /tmp 17616 MB (97% inode=99%): [19:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.04, 33.47, 32.90 [19:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91865 MB (15% inode=99%): [19:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291222 MB (5% inode=29%): [19:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:51:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:53:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [20:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:15:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.88, 30.33, 31.57 [20:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91664 MB (15% inode=99%): [20:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289757 MB (5% inode=29%): [20:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:51:02] Load avg. on willow is WARNING: WARNING - load average: 5.00, 10.01, 19.49 [20:51:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:54:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:57:02] Load avg. on willow is OK: OK - load average: 4.24, 5.93, 14.46 [21:04:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:21:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91530 MB (15% inode=99%): [21:26:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291249 MB (5% inode=29%): [21:27:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:48:07] Christian Thiele * Re: [Toolserver-l] Cron on willow [21:51:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:03:08] DaB. * [Toolserver-announce] New Rule: SGE-constraint for bots [22:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [22:05:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91366 MB (15% inode=99%): [22:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:26:07] Liangent * Re: [Toolserver-l] New Rule: SGE-constraint for bots [22:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288590 MB (5% inode=29%): [22:27:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 92852 MB (7% inode=99%): [22:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:07] DaB. * Re: [Toolserver-l] New Rule: SGE-constraint for bots [22:51:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B: [23:05:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:18:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:21:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91112 MB (14% inode=99%): [23:23:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:26:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290356 MB (5% inode=29%): [23:27:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:51:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:54:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7