[00:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:02:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:04:12] Free Memory on turnera is WARNING: WARNING - 5.7% (479304 kB) free! [00:07:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:11:49] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:14:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:14:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:15:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120277 MB (19% inode=99%): [00:17:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:18:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:21:12] Free Memory on turnera is OK: OK - 7.1% (591916 kB) free. [00:21:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:22:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:26:20] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:27:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:31:12] Free Memory on turnera is WARNING: WARNING - 5.8% (488024 kB) free! [00:32:12] Free Memory on turnera is OK: OK - 7.1% (594484 kB) free. [00:40:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:02:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:07:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:11:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:14:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:14:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:15:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120130 MB (19% inode=99%): [01:17:12] Free Memory on turnera is WARNING: WARNING - 6.3% (526084 kB) free! [01:17:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:18:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:21:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:23:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:27:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:28:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:40:11] Free Memory on turnera is OK: OK - 7.1% (595160 kB) free. [01:40:20] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:55:49] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 123566 MB (12% inode=99%): [02:02:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:04:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:08:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:11:49] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:14:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:16:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120035 MB (19% inode=99%): [02:18:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:18:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:23:12] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:27:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:28:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:31:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:36:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:40:21] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:02:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:04:12] Free Memory on turnera is WARNING: WARNING - 6.6% (549356 kB) free! [03:04:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:08:12] Free Memory on turnera is OK: OK - 7.4% (618292 kB) free. [03:08:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:11:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:14:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:15:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:16:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119979 MB (19% inode=99%): [03:18:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:18:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:21:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:23:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:28:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:28:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:41:19] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:43:12] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (374332 kB) free! [03:55:12] Free Memory on turnera is WARNING: WARNING - 6.4% (532844 kB) free! [03:59:13] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (356540 kB) free! [04:02:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:04:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:09:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:11:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:14:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:17:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119919 MB (19% inode=99%): [04:18:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:18:35] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:21:34] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:23:15] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:28:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:29:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:41:27] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:02:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:04:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:10:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:11:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:14:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:17:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119808 MB (19% inode=99%): [05:18:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:18:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:24:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:28:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:30:06] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:41:28] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:02:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:04:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:07:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422908 kB) free! [06:10:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:11:53] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:14:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:15:26] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:16:27] Free Memory on turnera is OK: OK - 11.1% (933212 kB) free. [06:17:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119704 MB (19% inode=99%): [06:18:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:18:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:24:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:28:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:29:26] Free Memory on turnera is WARNING: WARNING - 6.1% (513364 kB) free! [06:30:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:36:26] Free Memory on turnera is OK: OK - 7.4% (623904 kB) free. [06:42:27] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:51:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:02:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:04:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:10:27] Free Memory on turnera is WARNING: WARNING - 5.4% (454976 kB) free! [07:10:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:11:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:14:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:15:28] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:18:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119556 MB (19% inode=99%): [07:18:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:18:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:24:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:27] Free Memory on turnera is OK: OK - 7.5% (626664 kB) free. [07:28:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:30:06] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:33:07] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 56019 MB (5% inode=97%): [07:42:27] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:03:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:04:27] Free Memory on turnera is WARNING: WARNING - 6.2% (517032 kB) free! [08:04:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:07:14] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196072 MB (20% inode=99%): [08:11:28] Free Memory on turnera is OK: OK - 7.4% (619372 kB) free. [08:11:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:11:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:14:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:15:28] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:18:06] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119397 MB (19% inode=99%): [08:18:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:18:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:24:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:28:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:30:06] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:42:32] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:56:26] Free Memory on turnera is WARNING: WARNING - 6.3% (531808 kB) free! [08:59:27] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (391028 kB) free! [09:00:28] Free Memory on turnera is WARNING: WARNING - 5.9% (494128 kB) free! [09:03:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:04:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:11:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:11:53] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:14:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:16:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:18:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119208 MB (19% inode=99%): [09:18:29] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:19:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:20:29] @replag [09:20:30] Merlissimo: s1-rr-a-wd: 1m 57s [+0.00 s/s]; s1-user-wd: 1m 57s [+0.00 s/s]; s2-user-wd: 1m 57s [+0.00 s/s]; s4-user-wd: 1m 57s [+0.00 s/s]; s5-user-wd: 1m 57s [+0.00 s/s]; s6-rr-a: 3m 45s [+0.00 s/s]; s6-user: 3m 45s [+0.00 s/s]; s7-rr-a: 10s [-0.00 s/s] [09:20:31] Merlissimo: s7-user: 10s [-0.00 s/s] [09:24:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:28:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:30:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:42:27] Free Memory on turnera is WARNING: WARNING - 6.6% (550624 kB) free! [09:43:27] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:49:27] Free Memory on turnera is OK: OK - 7.1% (597396 kB) free. [09:51:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:57:28] Free Memory on turnera is WARNING: WARNING - 6.4% (537984 kB) free! [09:59:27] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (389148 kB) free! [10:03:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:04:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:11:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:12:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:14:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:16:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:18:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118975 MB (19% inode=99%): [10:19:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:20:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:21:28] Free Memory on turnera is CRITICAL: CRITICAL - 3.8% (320732 kB) free! [10:24:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:27:27] Free Memory on turnera is WARNING: WARNING - 5.2% (435068 kB) free! [10:28:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:30:06] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:32:08] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 26457 MB (6% inode=99%): [10:43:27] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:47:18] @replag [10:47:19] Merlissimo: s1-rr-a-wd: 1m 43s [-0.00 s/s]; s1-user-wd: 1m 43s [-0.00 s/s]; s2-user: 22s [+0.00 s/s]; s2-user-wd: 1m 43s [-0.00 s/s]; s3-rr-a: 15m 1s [+0.02 s/s]; s3-user: 15m 1s [+0.02 s/s]; s4-user-wd: 1m 43s [-0.00 s/s]; s5-user-wd: 1m 43s [-0.00 s/s] [10:47:20] Merlissimo: s7-rr-a: 9m 45s [+0.11 s/s]; s7-user: 9m 45s [+0.11 s/s] [10:59:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (359412 kB) free! [11:01:28] Free Memory on turnera is WARNING: WARNING - 5.2% (436348 kB) free! [11:03:28] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422376 kB) free! [11:03:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:05:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:11:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:14:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:16:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:18:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 118685 MB (19% inode=99%): [11:20:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:20:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:21:35] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:23:27] Free Memory on turnera is WARNING: WARNING - 6.5% (542784 kB) free! [11:24:28] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:28:07] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1574 MB (12% inode=99%): [11:29:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:30:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:31:07] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 57543 MB (5% inode=97%): [11:44:27] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:55:06] /tmp on ortelius is OK: DISK OK - free space: /tmp 6483 MB (51% inode=99%): [11:58:27] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417280 kB) free! [12:03:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:05:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:06:27] Free Memory on turnera is WARNING: WARNING - 5.2% (435532 kB) free! [12:09:26] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (373848 kB) free! [12:12:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:15:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:16:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:17:07] /sql on ptolemy is OK: DISK OK - free space: /sql 181108 MB (30% inode=99%): [12:20:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:20:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:21:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:24:28] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:29:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:30:07] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:44:27] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:48:07] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 24.81, 44.79, 25.93 [12:53:08] Load avg. on ortelius is WARNING: WARNING - load average: 5.36, 19.41, 19.85 [12:55:27] Free Memory on turnera is WARNING: WARNING - 6.0% (505812 kB) free! [12:59:07] Load avg. on ortelius is OK: OK - load average: 3.03, 8.17, 14.45 [12:59:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.8% (317948 kB) free! [13:02:27] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [13:02:35] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [13:03:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:04:13] /sql on thyme is OK: DISK OK - free space: /sql 196444 MB (20% inode=99%): [13:05:27] Free Memory on turnera is WARNING: WARNING - 5.3% (446860 kB) free! [13:05:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:06:27] MySQL on z-dat-s6-a is OK: Uptime: 610 Threads: 5 Questions: 2330 Slow queries: 0 Opens: 153 Flush tables: 1 Open tables: 144 Queries per second avg: 3.819 [13:06:36] MySQL slave on z-dat-s6-a is OK: Uptime: 621 Threads: 4 Questions: 4309 Slow queries: 1 Opens: 161 Flush tables: 1 Open tables: 152 Queries per second avg: 6.938 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1037 [13:08:06] /sql on rosemary is OK: DISK OK - free space: /sql 124166 MB (12% inode=99%): [13:12:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:14:27] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (404620 kB) free! [13:14:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:15:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:16:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:20:27] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:20:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:25:27] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:30:12] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:30:28] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:33:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:33:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:34:09] Free Memory on turnera is OK: OK - 7.2% (600264 kB) free. [13:36:59] Environment IPMI on thyme is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:37:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:02] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:44:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:45:58] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [13:50:38] Free Memory on turnera is WARNING: WARNING - 5.5% (463796 kB) free! [13:55:04] DaB. * [Toolserver-announce] sql-s1-rr will be switched this evening [13:59:58] @replag [13:59:59] Danny_B|backup: s1-rr-a-wd: 57s [+0.01 s/s]; s1-user-wd: 1m 3s [+0.01 s/s]; s2-user-wd: 1m 3s [+0.00 s/s]; s3-rr-a: 6m 30s [+0.07 s/s]; s3-user: 6m 30s [+0.07 s/s]; s4-user-wd: 57s [+0.01 s/s]; s5-user-wd: 1m 3s [+0.00 s/s]; s7-rr-a: error [14:00:00] Danny_B|backup: s7-user: error [14:00:16] hmmm what's up with s7? [14:12:18] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:38:27] RAID on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:39:28] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:39:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:39:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:40:48] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [14:40:48] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [14:44:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:45:59] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [14:55:48] MySQL on z-dat-s6-a is OK: Uptime: 982 Threads: 2 Questions: 1430 Slow queries: 0 Opens: 111 Flush tables: 1 Open tables: 103 Queries per second avg: 1.456 [14:59:50] @replag [14:59:51] Merlissimo: s1-rr-a-wd: 36s [-0.01 s/s]; s1-user-wd: 27s [-0.01 s/s]; s2-user: 10s [-0.00 s/s]; s2-user-wd: 41s [-0.01 s/s]; s3-rr-a: error; s3-user: error; s4-user-wd: 37s [-0.01 s/s]; s5-user-wd: 41s [-0.01 s/s] [14:59:52] Merlissimo: s6-rr-a: 24m 55s [+0.12 s/s]; s6-user: 24m 55s [+0.12 s/s]; s7-rr-a: error; s7-user: error [15:00:48] MySQL on z-dat-s3-a is OK: Uptime: 1283 Threads: 3 Questions: 55 Slow queries: 0 Opens: 42 Flush tables: 1 Open tables: 34 Queries per second avg: 0.42 [15:01:39] @replag [15:01:41] Merlissimo: s1-rr-a-wd: 17s [-0.18 s/s]; s1-user-wd: 17s [-0.09 s/s]; s2-user-wd: 18s [-0.21 s/s]; s3-rr-a: 1h 60s [+0.88 s/s]; s3-user: 1h 1m 2s [+0.88 s/s]; s4-user-wd: 19s [-0.16 s/s]; s5-user-wd: 19s [-0.20 s/s]; s6-rr-a: 9m 38s [-8.32 s/s] [15:01:42] Merlissimo: s6-user: 9m 38s [-8.32 s/s]; s7-rr-a: error; s7-user: error [15:12:18] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:40:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:40:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:44:59] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:46:58] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [16:12:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:14:11] if I open a listening socket on one of the Toolserver servers, it will be available to other machines inside the cluster but not the outside world, right? [16:15:48] (which is how I want it to work, just want to check) [16:40:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:41:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:44:59] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:46:58] MySQL on z-dat-s7-a is CRITICAL: Lost connection to MySQL server at reading initial communication packet, system error: 131 [17:12:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:40:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:41:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:44:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:46:58] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:12:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:44:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:46:58] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:12:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:19:44] hello all [19:23:04] sql-s1-rr ist switched now [19:31:19] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [19:34:52] @replag [19:34:53] Danny_B|backup: s1-rr-a: error; s1-rr-a-wd: error; s1-user-wd: 1m 26s [+0.00 s/s]; s2-user-wd: 1m 32s [+0.00 s/s]; s3-rr-a: 50s [-0.14 s/s]; s3-user: 50s [-0.14 s/s]; s4-user-wd: 1m 26s [+0.00 s/s]; s5-user-wd: 1m 32s [+0.00 s/s] [19:34:53] Danny_B|backup: s7-rr-a: error; s7-user: error [19:35:03] DaBPunkt: what's with s7? [19:42:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:42:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:44:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:46:59] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [20:12:19] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:31:19] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [20:42:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:42:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:44:58] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:46:59] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [20:57:45] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:57:50] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [20:57:51] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:03:22] Danny_B|backup: looks like its mysql is broken [21:03:46] :-( [21:04:18] maybe something got blocked. I will reboot this instance [21:05:11] thx [21:12:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:18:22] Danny_B|backup: the reboot didn't help. nosy looks where the problem is and if we can fix it [21:19:20] thx [21:21:50] MySQL on z-dat-s7-a is OK: Uptime: 37 Threads: 3 Questions: 157 Slow queries: 0 Opens: 64 Flush tables: 1 Open tables: 57 Queries per second avg: 4.243 [21:32:01] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [21:57:00] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:57:40] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:57:51] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:12:30] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:27:11] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:27:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:27:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:32:28] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [22:49:43] @replag [22:49:44] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user-wd: 1m 26s [-]; s2-user-wd: 1m 26s [-0.00 s/s]; s3-rr-a: 6m 9s [+0.03 s/s]; s3-user: 6m 9s [+0.03 s/s]; s4-user-wd: 1m 26s [-]; s5-user-wd: 1m 26s [-0.00 s/s] [22:49:45] DaBPunkt: s7-rr-a: 6h 59m 54s [+0.67 s/s]; s7-user: 6h 59m 54s [+0.67 s/s] [22:50:30] \o/ \o/ \o/ [22:50:36] nice job guys [23:26:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:07] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:27:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:32:27] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146)