[00:08:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:08:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:09:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:10:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:11:13] @replag [00:11:14] DaBPunkt: s1-rr-a-wd: 1m 58s [+0.00 s/s]; s1-user: 20s [-0.02 s/s]; s1-user-wd: 1m 58s [+0.00 s/s]; s2-user: 37s [+0.01 s/s]; s2-user-wd: 1m 58s [+0.00 s/s]; s4-user-wd: 1m 58s [+0.00 s/s]; s5-user-wd: 1m 58s [+0.00 s/s] [00:12:19] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:13:42] nacht ts [00:14:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:16:01] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:16:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:18:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:21:02] @replag [00:21:02] Alchimista: s1-rr-a-wd: 1m 12s [-0.08 s/s]; s1-user-wd: 1m 12s [-0.08 s/s]; s2-user-wd: 1m 12s [-0.08 s/s]; s3-rr-a: 1m 3s [+0.02 s/s]; s3-user: 1m 3s [+0.02 s/s]; s4-user-wd: 1m 12s [-0.08 s/s]; s5-user-wd: 1m 12s [-0.08 s/s] [00:27:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:30:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:30:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:03:29] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 41.09, 25.91, 14.54 [01:08:20] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:08:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:09:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:10:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:12:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:13:28] Load avg. on ortelius is WARNING: WARNING - load average: 9.30, 20.03, 19.58 [01:14:06] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:15:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:16:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:18:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:27:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:30:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:30:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:31:31] Load avg. on ortelius is OK: OK - load average: 11.38, 13.21, 14.89 [01:35:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (384620 kB) free! [01:54:29] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 124471 MB (12% inode=99%): [01:55:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [02:07:57] Free Memory on turnera is WARNING: WARNING - 5.1% (427716 kB) free! [02:08:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420976 kB) free! [02:09:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:10:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:10:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:12:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:14:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:15:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:18:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:19:29] Load avg. on ortelius is WARNING: WARNING - load average: 15.41, 15.18, 14.69 [02:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:22:28] Load avg. on ortelius is OK: OK - load average: 12.83, 14.68, 14.61 [02:27:39] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:30:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:30:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:55:58] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [03:08:18] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:08:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (394272 kB) free! [03:09:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:10:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:10:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:10:57] Free Memory on turnera is OK: OK - 8.8% (738648 kB) free. [03:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:14:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:16:02] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:17:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:19:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:21:31] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:23:01] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (401908 kB) free! [03:24:00] @replag [03:24:00] Danny_B|backup: s1-rr-a-wd: 38s [-0.00 s/s]; s1-user-wd: 38s [-0.00 s/s]; s2-user-wd: 38s [-0.00 s/s]; s3-rr-a: 1h 35m 15s [+0.52 s/s]; s3-user: 1h 35m 15s [+0.52 s/s]; s4-user-wd: 38s [-0.00 s/s]; s5-user-wd: 39s [-0.00 s/s]; s6-rr-a: 12s [+0.00 s/s] [03:24:00] Danny_B|backup: s6-user: 12s [+0.00 s/s] [03:24:17] @replag all [03:24:18] Danny_B|backup: s1-rr-a: 0s [-0.00 s/s]; s1-rr-a-wd: 48s [+0.47 s/s]; s1-user: 2s [-0.00 s/s]; s1-user-c: 4s [-0.00 s/s]; s1-user-wd: 49s [+0.52 s/s]; s2-user: 2s [-0.00 s/s]; s2-user-c: 5s [-0.00 s/s]; s2-user-wd: 49s [+0.52 s/s] [03:24:19] Danny_B|backup: s3-rr-a: 1h 35m 37s [+1.04 s/s]; s3-user: 1h 35m 37s [+1.04 s/s]; s4-rr-a: 5s [-0.00 s/s]; s4-user: 5s [-0.00 s/s]; s4-user-wd: 49s [+0.52 s/s]; s5-rr-a: 2s [-0.00 s/s]; s5-user: 2s [-0.00 s/s]; s5-user-c: 5s [-0.00 s/s] [03:24:20] Danny_B|backup: s5-user-wd: 49s [+0.47 s/s]; s6-rr-a: 6s [-0.28 s/s]; s6-user: 6s [-0.28 s/s]; s7-rr-a: 3s [+0.00 s/s]; s7-user: 3s [+0.00 s/s] [03:27:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:30:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:37:58] Free Memory on turnera is WARNING: WARNING - 5.1% (423472 kB) free! [03:38:57] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (409868 kB) free! [03:55:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [04:00:58] Free Memory on turnera is WARNING: WARNING - 7.0% (582836 kB) free! [04:02:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (361772 kB) free! [04:09:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:09:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:10:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:10:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:12:58] Free Memory on turnera is WARNING: WARNING - 5.4% (448916 kB) free! [04:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:14:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:15:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:17:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:19:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:27:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:30:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:30:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:51:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:56:38] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 55137 MB (5% inode=97%): [04:57:00] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [05:09:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:09:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:10:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:10:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:14:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:16:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:17:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:19:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:27:37] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:30:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:30:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:43:39] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 44.01, 33.29, 18.27 [05:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:57:59] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:09:18] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:09:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:10:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:10:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:14:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:16:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:17:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:19:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:27:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:30:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:30:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:33:57] Free Memory on turnera is WARNING: WARNING - 5.1% (428352 kB) free! [06:44:40] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 43.26, 42.06, 39.90 [06:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:57:57] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:58:57] Free Memory on turnera is CRITICAL: CRITICAL - 3.5% (293676 kB) free! [07:00:57] Free Memory on turnera is WARNING: WARNING - 5.8% (488440 kB) free! [07:02:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422360 kB) free! [07:09:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:10:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:10:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:11:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:13:19] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:14:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:16:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:18:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:19:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:22:39] Load avg. on ortelius is WARNING: WARNING - load average: 3.41, 6.13, 19.69 [07:22:56] Free Memory on turnera is WARNING: WARNING - 5.1% (428456 kB) free! [07:24:02] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422948 kB) free! [07:25:02] Free Memory on turnera is WARNING: WARNING - 5.1% (427520 kB) free! [07:27:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:28:38] Load avg. on ortelius is OK: OK - load average: 2.53, 3.89, 14.22 [07:31:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:31:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:57:57] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [07:58:57] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (326436 kB) free! [08:05:57] Free Memory on turnera is WARNING: WARNING - 5.6% (467356 kB) free! [08:06:58] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196711 MB (20% inode=99%): [08:09:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:10:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:10:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:11:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:13:21] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:14:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:16:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:18:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:19:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:28:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:31:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:32:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:54:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418464 kB) free! [08:55:57] Free Memory on turnera is WARNING: WARNING - 5.1% (428620 kB) free! [08:57:57] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [08:59:01] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (239504 kB) free! [09:09:20] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:10:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:11:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:15:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:17:01] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:19:01] Free Memory on turnera is WARNING: WARNING - 5.3% (444028 kB) free! [09:19:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:19:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:28:39] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:29:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (419880 kB) free! [09:30:58] Free Memory on turnera is WARNING: WARNING - 5.3% (440576 kB) free! [09:31:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:32:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:05:11] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20181 [10:05:11] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (249216 kB) free! [10:05:58] Free Memory on turnera is WARNING: WARNING - 5.3% (443724 kB) free! [10:09:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:10:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:11:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:11:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:13:56] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422076 kB) free! [10:15:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:18:01] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:19:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:19:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:21:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:21:57] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3152 [10:24:58] MySQL slave on z-dat-s3-a is OK: Uptime: 86117 Threads: 10 Questions: 62325674 Slow queries: 1048 Opens: 943576 Flush tables: 1 Open tables: 16384 Queries per second avg: 723.732 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1558 [10:28:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:30:39] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 56999 MB (5% inode=97%): [10:31:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:31:30] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 26968 MB (6% inode=99%): [10:33:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:33:58] Free Memory on turnera is WARNING: WARNING - 5.7% (477636 kB) free! [10:58:57] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (245572 kB) free! [11:05:57] Free Memory on turnera is WARNING: WARNING - 5.3% (445724 kB) free! [11:09:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:10:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:11:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:11:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:15:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:17:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:19:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:19:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:21:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:29:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:31:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:33:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:36:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418448 kB) free! [11:39:00] Free Memory on turnera is WARNING: WARNING - 5.1% (423752 kB) free! [11:53:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422900 kB) free! [12:09:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:10:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:11:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:11:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:13:20] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:15:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:17:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:19:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:20:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:21:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:22:15] @replag all [12:22:15] Danny_B|backup: s1-rr-a: 0s [-]; s1-rr-a-wd: 59s [+0.00 s/s]; s1-user: 0s [-0.00 s/s]; s1-user-c: 1s [-0.00 s/s]; s1-user-wd: 60s [+0.00 s/s]; s2-user: 6s [+0.00 s/s]; s2-user-c: 1s [-0.00 s/s]; s2-user-wd: 60s [+0.00 s/s] [12:22:19] Danny_B|backup: s3-rr-a: 1m 28s [-0.18 s/s]; s3-user: 1m 28s [-0.18 s/s]; s4-rr-a: 1s [-0.00 s/s]; s4-user: 1s [-0.00 s/s]; s4-user-wd: 60s [+0.00 s/s]; s5-rr-a: 2s [-]; s5-user: 2s [-]; s5-user-c: 1s [-0.00 s/s] [12:22:19] Danny_B|backup: s5-user-wd: 60s [+0.00 s/s]; s6-rr-a: 1s [-0.00 s/s]; s6-user: 1s [-0.00 s/s]; s7-rr-a: 5s [+0.00 s/s]; s7-user: 5s [+0.00 s/s] [12:22:25] weird [12:29:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:32:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:32:57] Free Memory on turnera is WARNING: WARNING - 5.1% (425420 kB) free! [12:33:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:33:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (419280 kB) free! [12:35:57] Free Memory on turnera is WARNING: WARNING - 5.2% (433544 kB) free! [12:42:57] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (413992 kB) free! [13:09:16] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:10:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:11:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:11:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:15:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:17:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:19:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:20:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:30:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:32:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:33:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:55:28] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 123993 MB (12% inode=99%): [14:09:16] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:10:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:11:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:11:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:13:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:15:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:17:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:18:57] Free Memory on turnera is WARNING: WARNING - 5.2% (436816 kB) free! [14:19:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:20:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:30:39] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:32:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:33:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:59:00] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (242072 kB) free! [15:05:58] Free Memory on turnera is WARNING: WARNING - 5.3% (444072 kB) free! [15:09:18] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:10:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:11:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:11:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:11:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (421824 kB) free! [15:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:15:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:18:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:19:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:20:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:31:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:32:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:34:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:41:57] Free Memory on turnera is WARNING: WARNING - 5.3% (443376 kB) free! [15:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:58:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (237064 kB) free! [16:05:57] Free Memory on turnera is WARNING: WARNING - 5.3% (443256 kB) free! [16:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:10:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:11:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:11:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:13:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417152 kB) free! [16:16:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:18:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:19:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:20:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:22:00] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:31:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:31:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:32:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:34:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:56:39] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 56186 MB (5% inode=97%): [17:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:10:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:12:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:13:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:16:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:18:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:20:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:21:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:31:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:32:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:34:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:51:43] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:51:43] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:52:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:52:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:55:38] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:38] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:59:48] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:00:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:09:49] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:10:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:10:46] apmon_ has anything happened to the coastlien data on the osm db? [18:11:01] not that I know of [18:11:17] I'm loosing land in WMA (if I refetch JSON tiles from the server they do not contain any landmass polygons) [18:11:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:11:26] the world is under water!! [18:11:38] climate change? [18:12:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:12:28] Hopefully climate change doesn't put the entire world under water... ;-) [18:12:53] The tables still seem to exist in the db, and I am not aware of anything writing to those tables, so they shouldn't change [18:13:11] hm, te error is in Dubai only [18:13:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:13:18] wtf? [18:14:00] When I tested KothicJS the other day, also everything was under water, but I assumed that was because KothicJS didn't read the coastlines correctly [18:16:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:19:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:21:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:32:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:34:50] I'm checking other areas [18:34:56] Free Memory on turnera is WARNING: WARNING - 5.2% (432996 kB) free! [18:35:03] so far only Dubai is screwed up [18:36:55] hello all [18:36:55] @replag [18:36:56] DaBPunkt: s1-rr-a-wd: 46s [-0.00 s/s]; s1-user-wd: 47s [-0.00 s/s]; s2-user-wd: 47s [-0.00 s/s]; s3-rr-a: 2m 13s [+0.01 s/s]; s3-user: 2m 13s [+0.01 s/s]; s4-user-wd: 47s [-0.00 s/s]; s5-user-wd: 47s [-0.00 s/s] [18:37:53] dschwen: In case you are interested to have a look at KothicJS, you can point your web browser at port 3857 on willow. [18:39:07] I hope to get those javascript tiles running through the standard tirex / mod_tile setup soon, so that one can use them. [18:40:01] I wonder if you might be able to use those json tiles for WMA as well, but I suspect the main issue might be that you are using a different projection. [18:40:24] yeah [18:41:01] It then has the full caching, load and queing managment as well as expiry of tirex / mod_tile [18:41:47] ha [18:42:03] I think one of the coastline supertiles is missing near dubai [18:42:37] a big square is warter that should have been land [18:42:47] can you reimport by any chance? [18:43:17] DaBPunkt: The new mod_tile.so that hopefully works is at ~osm/tools2/mod_tile.so [18:43:45] But I have to go now for about an hour or so. [18:43:48] we can try it in 35min [18:43:54] ok, 1h is good too [18:43:57] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417380 kB) free! [18:46:57] Free Memory on turnera is WARNING: WARNING - 5.1% (427088 kB) free! [18:53:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:54:59] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420092 kB) free! [18:55:13] no tiles show up in KothicJS mode [18:59:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:01:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:08:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127206 MB (20% inode=99%): [19:10:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:10:20] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:12:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:12:16] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:16:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:19:58] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:21:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:21:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:32:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:49:20] DaBPunkt: I am around now for a while, in case you have the time [19:51:57] sure [19:53:01] As long as you have a backup of old mod_tile.so, there shouldn't be any problem rolling back in case things don't work,? [19:53:47] the mod_tile.so I have compiled is 32bit. Is apache compiled as 32bit as well? [19:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:55:30] "Cannot load /opt/ts/apache/2.2/modules/mod_tile.so into server: ld.so.1: httpd: fatal: relocation error: file /opt/ts/apache/2.2/modules/mod_tile.so: symbol vasprintf: referenced symbol not found" [19:55:48] I will switch back to the old one [19:56:06] Thanks. [19:56:16] Do you know what might cause that? [19:56:52] old versions runs again [19:58:44] an ldd for the new mod_tile indicates it is trying to link to libgcc_s.so.1 => /opt/ts/lib/libgcc_s.so.1 which the old one didn't do [19:59:09] I wonder if the relocation error is coming from that [19:59:12] the apache is 32bit AFAIS [20:00:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:01:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:02:41] it looks like colaris doesn't know about vasprintf [20:03:05] I guess I'll try and get rid of vasprintf in the source code :-S [20:03:31] It is only used once in the error reporting code [20:03:34] apmon_: http://blogs.everycity.co.uk/alasdair/2011/07/vasprintf-and-asprintf-on-solaris-10/ [20:03:52] but you can remove it too of corse [20:04:03] (I wonder why and how it compiled…) [20:05:07] it compiled with gcc rather than sun compiler [20:05:07] I see a big SQL error here: http://toolserver.org/~quentinv57/tools/sulinfo.php?username=Username [20:05:10] anyone aware? [20:05:23] which is why it probably is linked to libgcc_s.so.1 [20:06:57] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196056 MB (20% inode=99%): [20:07:28] AlexJFox: tell the tool-author, the server looks fine for me [20:07:38] okey doke [20:07:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126930 MB (20% inode=99%): [20:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:10:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:12:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:13:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:13:22] DaBPunkt: Do you know if vsnprintf exists? Then it shouldn't be difficult to replace vasprintf with vsnprintf and just allocate and free the string manually [20:14:30] google says: yes [20:15:58] Free Memory on turnera is WARNING: WARNING - 5.2% (433688 kB) free! [20:16:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:20:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:21:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:21:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420612 kB) free! [20:24:20] DaBPunkt: I have now replaced vasprintf with vsnprintf and a preallocated string. [20:24:51] In case you feel like trying again, a new compile is in ~osm/tools2/mod_tile.so [20:24:57] Free Memory on turnera is WARNING: WARNING - 5.1% (426144 kB) free! [20:25:37] There are probably more issues down the line, but I am not sure how to check for portability issues, other than trying to load the module [20:26:12] ok, wait a moment [20:27:25] looks good [20:27:46] it loaded? [20:27:48] some debug-output in the log about loadings [20:27:53] apmon_: yes [20:28:05] Let me check that things actually work in that case [20:28:09] ok [20:29:40] It is doing something odd [20:29:56] It looks like the module is working, but it is returing 404 instead of data [20:30:17] Do you know where the logs are? [20:30:34] [Wed Nov 07 20:30:22 2012] [notice] [client 10.24.1.20] Failed to connect to renderer, referer: http://toolserver.org/~kolossos/openlayers/embed.html?layer=mapnik&bbox=8.038904797927612,43.885767952148434,8.127761202072387,43.94756604785156&marker=43.916667,8.083333 [20:30:45] apmon_: /var/http/ [20:31:04] /var/log/http asorry [20:31:32] DaBPunkt: It looks like it is using the wrong tile directory to look for the data [20:31:52] is the dir hard-coded? [20:32:00] It is using the /var/lib/mod_tile/ directory rather than the /tiles directory [20:32:35] It shouldn't be, but I need to check where it is configured [20:32:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:32:57] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (406252 kB) free! [20:33:30] created a symlink [20:35:14] Great, looks better now [20:35:21] [Wed Nov 07 20:35:06 2012] [notice] [client 10.24.1.20] Failed to connect to renderer [20:35:22] [Wed Nov 07 20:35:06 2012] [warn] [client 10.24.1.20] socket connect failed for: /home/project/o/s/m/osm/tirex/var/run/tirex.sock with reason: Connection refused [20:35:28] but the log is still flooded [20:36:01] but it was also today in the morning, so lets say that's "normal" [20:36:01] Yes, those are the unix domain socket issues that we have been having all along [20:36:29] For some reason a good proportion of unix domain socket connects get a connection refused [20:36:43] just repeating it often enough and eventually it will succeed [20:37:17] how great… [20:37:20] But I haven't found a pattern what determins the probability of success for the connect [20:42:42] I think the "ModTileTileDir" directive in the apache site configuration should set the tile directory. It has a hard coded default, which I guess was used [20:43:04] apmon_ , you guys look pretty busy, but if you have some time, could you please do a reimport of the coastline data? [20:44:05] dschwen: I'll try and do that later today, if you don't mind. Remind me again should I forget. [20:47:31] sure, thanks! [20:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:00:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:01:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:08:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126720 MB (20% inode=99%): [21:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:11:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:12:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:13:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:13:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:16:28] DaBPunkt: Do you know where the apache site config is on ptolemy? [21:16:49] /etc/opt/ts/apache/2.2/ [21:17:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:20:47] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:21:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:32:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:00:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:01:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:07:00] @replag all [22:07:00] Danny_B|backup: s1-rr-a: 1s [+0.00 s/s]; s1-rr-a-wd: 1m 8s [+0.00 s/s]; s1-user: 1s [+0.00 s/s]; s1-user-c: 1s [-]; s1-user-wd: 1m 8s [+0.00 s/s]; s2-user: 2s [-0.00 s/s]; s2-user-c: 1s [-]; s2-user-wd: 1m 8s [+0.00 s/s] [22:07:01] Danny_B|backup: s3-rr-a: 5m 44s [+0.02 s/s]; s3-user: 5m 44s [+0.02 s/s]; s4-rr-a: 1s [-]; s4-user: 1s [-]; s4-user-wd: 1m 8s [+0.00 s/s]; s5-rr-a: 4s [+0.00 s/s]; s5-user: 4s [+0.00 s/s]; s5-user-c: 1s [-] [22:07:02] Danny_B|backup: s5-user-wd: 1m 8s [+0.00 s/s]; s6-rr-a: 5s [+0.00 s/s]; s6-user: 5s [+0.00 s/s]; s7-rr-a: 2s [-0.00 s/s]; s7-user: 2s [-0.00 s/s] [22:08:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126463 MB (20% inode=99%): [22:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:11:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:12:26] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:13:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:13:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:17:07] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:19:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:21:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:31:28] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 27099 MB (6% inode=99%): [22:31:39] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 57319 MB (5% inode=97%): [22:33:48] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:51:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:00:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:02:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:06:39] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58581 MB (6% inode=97%): [23:06:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58592 MB (6% inode=97%): [23:07:39] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58242 MB (5% inode=97%): [23:07:39] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58242 MB (5% inode=97%): [23:08:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126264 MB (20% inode=99%): [23:09:39] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59420 MB (6% inode=97%): [23:09:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59489 MB (6% inode=97%): [23:10:17] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:11:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:12:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:13:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:13:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:17:10] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:20:57] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:21:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:29:40] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58181 MB (5% inode=97%): [23:29:40] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58181 MB (5% inode=97%): [23:34:47] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:54:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output