[00:00:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:02:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:08:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126030 MB (20% inode=99%): [00:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:11:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:11:38] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 18.46, 29.73, 17.24 [00:12:32] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:12:38] Load avg. on ortelius is WARNING: WARNING - load average: 7.72, 24.65, 16.23 [00:13:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:17:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:20:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:21:55] @replag [00:21:58] DaBPunkt: s1-rr-a-wd: 1m 43s [+0.00 s/s]; s1-user: 12m 41s [+0.09 s/s]; s1-user-c: 10m 14s [+0.08 s/s]; s1-user-wd: 1m 43s [+0.00 s/s]; s2-user: 18s [+0.00 s/s]; s2-user-wd: 1m 43s [+0.00 s/s]; s3-rr-a: 23s [-0.04 s/s]; s3-user: 23s [-0.04 s/s] [00:21:59] DaBPunkt: s4-user-wd: 1m 43s [+0.00 s/s]; s5-user-wd: 1m 43s [+0.00 s/s] [00:22:38] Load avg. on ortelius is OK: OK - load average: 2.20, 12.51, 14.69 [00:22:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:33:10] nacht ts [00:36:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:01:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:02:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:05:38] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2462 MB (18% inode=99%): [01:08:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125933 MB (20% inode=99%): [01:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:11:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:12:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:13:09] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:13:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:17:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:20:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:22:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:35:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:55:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 124038 MB (12% inode=99%): [02:01:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:02:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:05:39] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1890 MB (15% inode=99%): [02:09:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125860 MB (20% inode=99%): [02:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:11:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:12:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:13:09] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:16:39] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 282 MB (2% inode=99%): [02:18:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:21:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:22:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:36:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:41:57] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:46:58] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:01:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:02:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:10:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125743 MB (20% inode=99%): [03:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:11:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:12:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:14:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:16:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 715 MB (5% inode=99%): [03:18:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:21:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:23:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:36:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:48:58] Free Memory on turnera is WARNING: WARNING - 6.9% (576116 kB) free! [03:49:58] Free Memory on turnera is OK: OK - 7.1% (593932 kB) free. [03:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:01:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:03:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:10:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125600 MB (20% inode=99%): [04:10:28] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:11:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:12:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:14:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:17:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 831 MB (6% inode=99%): [04:18:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:21:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:22:01] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:23:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:36:59] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:01:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:03:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:10:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:10:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125504 MB (20% inode=99%): [05:12:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:12:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:13:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:14:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:17:37] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 571 MB (4% inode=99%): [05:18:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:22:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:23:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:37:59] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:54:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:01:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:03:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:10:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125405 MB (20% inode=99%): [06:11:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:12:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:12:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:13:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:14:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:17:41] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 894 MB (6% inode=99%): [06:18:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:22:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:23:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:37:59] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:55:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:01:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:04:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:05:08] Free Memory on turnera is WARNING: WARNING - 6.8% (566332 kB) free! [07:07:08] Free Memory on turnera is OK: OK - 7.1% (597160 kB) free. [07:10:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125249 MB (20% inode=99%): [07:11:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:12:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:13:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:14:09] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:14:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:17:38] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 544 MB (4% inode=99%): [07:19:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:22:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:23:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:30:46] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58590 MB (6% inode=97%): [07:31:47] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 57612 MB (5% inode=97%): [07:39:00] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:00:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:01:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:04:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:04:37] /tmp on ortelius is OK: DISK OK - free space: /tmp 8273 MB (63% inode=99%): [08:06:59] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196099 MB (20% inode=99%): [08:10:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125045 MB (20% inode=99%): [08:11:27] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:12:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:15:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:15:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:19:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:22:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:38:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:00:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:01:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:02:11] @replag [09:02:14] Merlissimo: s1-rr-a-wd: 1m 15s [-0.00 s/s]; s1-user-wd: 1m 15s [-0.00 s/s]; s2-user-wd: 1m 15s [-0.00 s/s]; s3-rr-a: 1m 37s [+0.00 s/s]; s3-user: 1m 37s [+0.00 s/s]; s4-user-wd: 1m 15s [-0.00 s/s]; s5-user-wd: 1m 15s [-0.00 s/s]; s6-rr-a: 13s [+0.00 s/s] [09:02:14] Merlissimo: s6-user: 13s [+0.00 s/s]; s7-rr-a: 4m 4s [+0.01 s/s]; s7-user: 4m 4s [+0.01 s/s] [09:05:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:10:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124838 MB (20% inode=99%): [09:11:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:13:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:15:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:19:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:23:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:38:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:51:13] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [09:51:20] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [10:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:01:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:05:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:09:13] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2613 [10:09:21] MySQL on z-dat-s3-a is OK: Uptime: 1455 Threads: 8 Questions: 109 Slow queries: 0 Opens: 71 Flush tables: 1 Open tables: 61 Queries per second avg: 0.74 [10:09:29] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1993 [10:11:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124568 MB (20% inode=99%): [10:11:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:15:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:15:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:17:13] Free Memory on turnera is WARNING: WARNING - 6.4% (537660 kB) free! [10:19:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:19:29] MySQL slave on z-dat-s6-a is OK: Uptime: 172187 Threads: 25 Questions: 37734703 Slow queries: 7649 Opens: 303423 Flush tables: 1 Open tables: 2695 Queries per second avg: 219.149 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1798 [10:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:24:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:25:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:28:12] Free Memory on turnera is OK: OK - 7.4% (616948 kB) free. [10:31:48] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 26552 MB (6% inode=99%): [10:39:15] MySQL slave on z-dat-s3-a is OK: Uptime: 3244 Threads: 12 Questions: 1182960 Slow queries: 78 Opens: 18764 Flush tables: 1 Open tables: 11077 Queries per second avg: 364.660 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1584 [10:39:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:01:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:05:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:11:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124313 MB (20% inode=99%): [11:11:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:15:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:20:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:21:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:24:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:25:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:30:12] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 55822 MB (5% inode=97%): [11:39:13] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:52:13] Free Memory on turnera is WARNING: WARNING - 5.4% (451884 kB) free! [11:59:12] Free Memory on turnera is CRITICAL: CRITICAL - 3.5% (290320 kB) free! [12:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:01:12] Free Memory on turnera is WARNING: WARNING - 5.4% (451672 kB) free! [12:01:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:05:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:11:39] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:12:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124091 MB (20% inode=99%): [12:13:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:15:11] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:15:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:20:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:25:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:31:12] Free Memory on turnera is WARNING: WARNING - 5.6% (466360 kB) free! [12:39:13] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:52:12] Free Memory on turnera is OK: OK - 7.1% (596684 kB) free. [13:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:04:34] @replag [13:04:34] Merlissimo: s1-rr-a-wd: 55s [-0.00 s/s]; s1-user: 26s [-0.02 s/s]; s1-user-wd: 55s [-0.00 s/s]; s2-user-wd: 56s [-0.00 s/s]; s4-user-wd: 56s [-0.00 s/s]; s5-user-wd: 56s [-0.00 s/s] [13:05:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:11:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:12:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123795 MB (20% inode=99%): [13:13:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:15:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:15:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:20:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:25:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:39:13] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:55:48] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 124102 MB (12% inode=99%): [14:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:02:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:06:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:11:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:12:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123515 MB (20% inode=99%): [14:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:15:11] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:15:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:20:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:25:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:32:12] Free Memory on turnera is WARNING: WARNING - 5.5% (458188 kB) free! [14:37:12] Free Memory on turnera is OK: OK - 7.2% (606504 kB) free. [14:39:21] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:52:13] Free Memory on turnera is WARNING: WARNING - 6.6% (549464 kB) free! [14:59:06] hello all [14:59:13] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418088 kB) free! [15:00:02] @replag [15:00:03] DaBPunkt: s1-rr-a-wd: 32s [-0.00 s/s]; s1-user: 20s [-0.00 s/s]; s1-user-wd: 32s [-0.00 s/s]; s2-user-wd: 26s [-0.00 s/s]; s3-rr-a: 1m 1s [-0.00 s/s]; s3-user: 1m 1s [-0.00 s/s]; s4-user-wd: 32s [-0.00 s/s]; s5-user-wd: 26s [-0.00 s/s] [15:00:09] DaBPunkt: hi. do you still need that space & space-grep files? [15:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:00:30] Danny_B|backup: pardon? [15:01:09] so it must have been nosy then [15:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:05:08] Marlen Caemmerer * [Toolserver-announce] Maintenance of wolfsbane and thyme [15:06:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:11:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123180 MB (20% inode=99%): [15:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:15:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:16:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:20:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:39:21] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:51:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:55:03] DaBPunkt: ich habe wegen der mail eben sge auf wolfsbane am 11.11 von 20-22 disabled, so dass in der Zeit dort kein job laufen wird [15:57:14] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420684 kB) free! [15:58:13] Free Memory on turnera is WARNING: WARNING - 5.2% (435488 kB) free! [15:59:13] Free Memory on turnera is CRITICAL: CRITICAL - 3.7% (309780 kB) free! [16:00:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:06:57] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:07:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:11:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:13:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:14:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122840 MB (20% inode=99%): [16:15:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:16:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:50:51] np [16:50:51] speaking about webservers... any eta for switch to apache? [16:50:51] as for authentication, I plan to send users tokens that are stored in their webbrowsers [16:50:52] dschwen: this year. I tried some setups this week, but was not successful yet [16:50:52] with those they can regain access to a previously uploaded watchlist if they have to reconnect [16:50:52] dschwen: that's ok [16:50:53] as long as I can paticipate as JohnDoe (with DaB.'s watchlist-data provided by me) it is ok [16:50:53] Danny_B|backup: this year. I tried some setups this week, but was not successful yet [16:50:53] danke schon [16:51:52] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:51:53] Danny_B|backup: np. BTW: If you do not have a "รถ" on your keyboard, use "oe" instead of "o". [16:52:40] * Danny_B|backup always writes without accents [16:52:42] on irc [16:52:53] because people have different encodings [16:53:44] Danny_B|backup: these are not accents [16:57:49] DaBPunkt, will 32bit SunOS binaries work on our webservers? [16:58:01] or 64? [16:58:39] 64bit should, 32bit may [16:58:58] udd, uname -a doesn't sayy anything about 64bit [16:59:03] it mentiones i386 [16:59:12] Free Memory on turnera is CRITICAL: CRITICAL - 3.8% (320508 kB) free! [16:59:16] on linus this means 32bit, but what do I know about Solaris [16:59:22] x [17:00:25] "Solaris uses a common code base for the platforms it supports: SPARC and i86pc (which includes both x86 and x86-64)." (a untrusty web-lexicon ;)) [17:00:45] (and yes, we run 64bit-versions) [17:01:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:03:38] node: fatal: libsocket.so.1: version 'SUNW_1.7' not found [17:03:48] ok, it does not seem as straight forward [17:04:12] Free Memory on turnera is WARNING: WARNING - 5.1% (426788 kB) free! [17:05:12] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420396 kB) free! [17:07:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:11:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:13:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:14:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122505 MB (20% inode=99%): [17:15:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:16:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:20:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:25:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:25:12] Free Memory on turnera is WARNING: WARNING - 5.9% (492912 kB) free! [17:26:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:34:25] Free Memory on turnera is OK: OK - 7.1% (595624 kB) free. [17:39:21] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:43:13] Free Memory on turnera is WARNING: WARNING - 6.2% (519916 kB) free! [17:59:12] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (391588 kB) free! [18:01:12] Free Memory on turnera is WARNING: WARNING - 5.7% (480876 kB) free! [18:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:07:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:09:13] Free Memory on turnera is OK: OK - 7.1% (595556 kB) free. [18:11:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:13:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:13:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:15:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122126 MB (20% inode=99%): [18:16:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:17:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:20:11] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:25:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:26:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:39:21] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:56:12] Free Memory on turnera is WARNING: WARNING - 5.5% (462120 kB) free! [18:59:12] Free Memory on turnera is CRITICAL: CRITICAL - 4.0% (335440 kB) free! [19:01:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:02:12] Free Memory on turnera is WARNING: WARNING - 5.1% (423872 kB) free! [19:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:07:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:11:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:15:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121811 MB (20% inode=99%): [19:16:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:17:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:20:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:26:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:26:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:32:12] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 55179 MB (5% inode=97%): [19:39:20] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:54:12] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (362756 kB) free! [19:56:13] Free Memory on turnera is WARNING: WARNING - 5.1% (425428 kB) free! [19:58:12] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (366496 kB) free! [20:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:02:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:07:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196905 MB (20% inode=99%): [20:07:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:11:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:13:06] @replag [20:13:07] Merlissimo: s1-rr-a-wd: 31s [-0.00 s/s]; s1-user-wd: 37s [+0.00 s/s]; s2-user-wd: 31s [+0.00 s/s]; s4-user-wd: 7m 37s [+0.02 s/s]; s5-user-wd: 32s [+0.00 s/s] [20:13:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:13:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:15:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121479 MB (20% inode=99%): [20:16:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:17:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:21:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:26:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:26:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:39:21] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:02:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:05:12] Free Memory on turnera is CRITICAL: CRITICAL - 4.0% (333460 kB) free! [21:07:13] Free Memory on turnera is WARNING: WARNING - 5.3% (441964 kB) free! [21:07:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:11:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:13:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:15:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121128 MB (19% inode=99%): [21:17:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:17:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:21:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:22:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:26:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:26:13] Free Memory on turnera is OK: OK - 7.4% (618384 kB) free. [21:26:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:39:21] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:02:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:04:13] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417688 kB) free! [22:05:12] Free Memory on turnera is WARNING: WARNING - 5.8% (483032 kB) free! [22:07:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:11:08] @replag [22:11:09] dschwen: s1-rr-a-wd: 1m 17s [+0.01 s/s]; s1-user-wd: 56s [+0.00 s/s]; s2-user-wd: 56s [+0.00 s/s]; s3-rr-a: 41s [-0.00 s/s]; s3-user: 41s [-0.00 s/s]; s4-user-wd: 1m 17s [-0.05 s/s]; s5-user-wd: 56s [+0.00 s/s]; s7-rr-a: 17s [-0.00 s/s] [22:11:10] dschwen: s7-user: 17s [-0.00 s/s] [22:11:49] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:13:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:15:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120788 MB (19% inode=99%): [22:15:12] Free Memory on turnera is OK: OK - 7.3% (610096 kB) free. [22:17:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:18:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:22:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:26:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:26:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:31:49] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 26228 MB (6% inode=99%): [22:39:20] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:51:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:02:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:02:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:05:12] Free Memory on turnera is WARNING: WARNING - 6.9% (576720 kB) free! [23:06:12] Free Memory on turnera is OK: OK - 7.5% (631676 kB) free. [23:07:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:11:49] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:13:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:15:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120514 MB (19% inode=99%): [23:17:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:18:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:22:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:26:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:26:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:30:12] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 54749 MB (5% inode=97%): [23:39:20] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:58:21] nacht ts [23:59:00] @replag [23:59:01] DaBPunkt: s1-rr-a-wd: 1m 36s [+0.00 s/s]; s1-user-wd: 1m 29s [+0.01 s/s]; s2-user-wd: 1m 29s [+0.01 s/s]; s3-rr-a: 1m 44s [+0.01 s/s]; s3-user: 1m 44s [+0.01 s/s]; s4-user-wd: 1m 29s [+0.00 s/s]; s5-user-wd: 1m 29s [+0.01 s/s]