[00:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:02:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:02:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:04:27] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:08:02] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84189 MB (13% inode=99%): [00:20:28] / on wolfsbane is WARNING: DISK WARNING - free space: / 6272 MB (20% inode=93%): [00:20:28] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6272 MB (20% inode=93%): [00:21:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:23:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:24:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [00:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:26:59] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [00:27:07] RAID on rosemary is CRITICAL: Connection refused by host [00:29:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [00:32:50] Load avg. on rosemary is CRITICAL: Connection refused by host [00:32:50] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [00:32:58] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [00:32:58] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [00:33:08] / on rosemary is CRITICAL: Connection refused by host [00:33:20] SMTP on rosemary is CRITICAL: Connection refused [00:33:28] SSH on rosemary is CRITICAL: Connection refused [00:33:28] /tmp on rosemary is CRITICAL: Connection refused by host [00:33:28] Environment IPMI on rosemary is CRITICAL: Connection refused by host [01:01:49] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:02:19] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:03:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:03:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:04:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:08:52] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:18:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84097 MB (13% inode=99%): [01:20:28] / on wolfsbane is WARNING: DISK WARNING - free space: / 6269 MB (20% inode=93%): [01:20:28] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6269 MB (20% inode=93%): [01:21:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:23:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:24:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [01:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:26:57] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [01:27:08] RAID on rosemary is CRITICAL: Connection refused by host [01:29:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [01:32:48] Load avg. on rosemary is CRITICAL: Connection refused by host [01:32:48] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [01:32:58] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [01:32:58] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [01:33:11] / on rosemary is CRITICAL: Connection refused by host [01:33:18] SMTP on rosemary is CRITICAL: Connection refused [01:33:28] SSH on rosemary is CRITICAL: Connection refused [01:33:28] /tmp on rosemary is CRITICAL: Connection refused by host [01:33:28] Environment IPMI on rosemary is CRITICAL: Connection refused by host [02:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:04:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:04:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:08:48] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:10:07] nosy: I can't connect to rosemary, is this expected? [02:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84003 MB (13% inode=99%): [02:20:30] / on wolfsbane is WARNING: DISK WARNING - free space: / 6256 MB (20% inode=93%): [02:20:30] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6256 MB (20% inode=93%): [02:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:23:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:23:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:24:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [02:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:26:58] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [02:27:13] RAID on rosemary is CRITICAL: Connection refused by host [02:29:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [02:32:48] Load avg. on rosemary is CRITICAL: Connection refused by host [02:32:48] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [02:33:02] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [02:33:02] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [02:33:08] / on rosemary is CRITICAL: Connection refused by host [02:33:18] SMTP on rosemary is CRITICAL: Connection refused [02:33:33] SSH on rosemary is CRITICAL: Connection refused [02:33:33] /tmp on rosemary is CRITICAL: Connection refused by host [02:33:33] Environment IPMI on rosemary is CRITICAL: Connection refused by host [03:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:02:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:02:42] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:03:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:04:29] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:06:30] / on wolfsbane is OK: DISK OK - free space: / 6291 MB (21% inode=93%): [03:09:03] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:11:28] /tmp on wolfsbane is OK: DISK OK - free space: / 6739 MB (22% inode=93%): [03:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83915 MB (13% inode=99%): [03:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:22:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:23:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:24:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [03:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:26:58] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [03:27:08] RAID on rosemary is CRITICAL: Connection refused by host [03:29:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [03:32:57] Load avg. on rosemary is CRITICAL: Connection refused by host [03:32:58] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [03:32:58] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [03:32:58] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [03:33:08] / on rosemary is CRITICAL: Connection refused by host [03:33:19] SMTP on rosemary is CRITICAL: Connection refused [03:33:28] SSH on rosemary is CRITICAL: Connection refused [03:33:28] /tmp on rosemary is CRITICAL: Connection refused by host [03:33:28] Environment IPMI on rosemary is CRITICAL: Connection refused by host [03:53:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:01:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:02:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:04:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:07:58] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:08:58] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83828 MB (13% inode=99%): [04:21:51] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:24:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [04:24:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:25:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:26:57] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [04:27:19] RAID on rosemary is CRITICAL: Connection refused by host [04:29:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [04:32:59] Load avg. on rosemary is CRITICAL: Connection refused by host [04:33:00] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [04:33:00] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [04:33:09] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [04:33:18] / on rosemary is CRITICAL: Connection refused by host [04:33:18] SMTP on rosemary is CRITICAL: Connection refused [04:33:28] /tmp on rosemary is CRITICAL: Connection refused by host [04:33:28] SSH on rosemary is CRITICAL: Connection refused [04:33:28] Environment IPMI on rosemary is CRITICAL: Connection refused by host [05:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:02:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:02:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:04:27] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:08:58] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:17:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83722 MB (13% inode=99%): [05:21:47] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:23:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:24:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [05:24:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:26:58] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [05:28:18] RAID on rosemary is CRITICAL: Connection refused by host [05:29:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [05:32:58] Load avg. on rosemary is CRITICAL: Connection refused by host [05:33:08] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [05:33:20] / on rosemary is CRITICAL: Connection refused by host [05:33:20] SMTP on rosemary is CRITICAL: Connection refused [05:33:28] SSH on rosemary is CRITICAL: Connection refused [05:33:28] /tmp on rosemary is CRITICAL: Connection refused by host [05:33:28] Environment IPMI on rosemary is CRITICAL: Connection refused by host [05:33:58] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [05:33:58] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [06:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:02:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:02:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:02:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:03:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:04:29] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:08:59] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83542 MB (13% inode=99%): [06:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:23:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:24:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [06:24:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:26:58] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [06:28:17] RAID on rosemary is CRITICAL: Connection refused by host [06:29:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [06:32:58] Load avg. on rosemary is CRITICAL: Connection refused by host [06:33:12] FMA on rosemary is CRITICAL: ERROR - unexpected output from snmpwalk [06:33:18] / on rosemary is CRITICAL: Connection refused by host [06:33:18] SMTP on rosemary is CRITICAL: Connection refused [06:33:27] SSH on rosemary is CRITICAL: Connection refused [06:33:28] /tmp on rosemary is CRITICAL: Connection refused by host [06:33:28] Environment IPMI on rosemary is CRITICAL: Connection refused by host [06:33:58] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [06:33:58] NTP on rosemary is CRITICAL: NTP CRITICAL: No response from NTP server [06:50:09] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/10:DOWN: 1 int NOK : CRITICAL [06:50:18] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:DOWN: 1 int NOK : CRITICAL [06:52:11] FC 0/10 [rosemary] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/10:UP:1 UP: OK [06:52:18] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/6:UP:1 UP: OK [07:01:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:02:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:02:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:02:40] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:04:28] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:08:57] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:12:07] MySQL on rosemary is OK: Uptime: 996 Threads: 46 Questions: 494 Slow queries: 15 Opens: 116 Flush tables: 1 Open tables: 108 Queries per second avg: 0.495 [07:15:18] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 11.638 second response time [07:18:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83358 MB (13% inode=99%): [07:21:48] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:22:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:24:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:24:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31524.000000 [07:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:27:08] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31445 [07:28:18] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:29:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29321.000000 [07:41:00] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 64874 MB (6% inode=97%): [07:41:08] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 64869 MB (6% inode=97%): [07:58:59] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199935 MB (20% inode=99%): [08:01:30] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:01:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:02:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:02:29] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 15.706 second response time [08:02:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:03:58] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:04:08] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.505 second response time [08:08:58] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:15:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:15:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:15:45] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:18:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83151 MB (13% inode=99%): [08:20:45] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2144 MB (14% inode=99%): [08:21:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:23:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:24:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:25:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32684.000000 [08:26:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:27:45] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:29:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:30:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32961.000000 [08:34:46] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1244 MB (8% inode=99%): [08:35:46] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1955 MB (13% inode=99%): [08:41:46] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30356 MB (7% inode=99%): [09:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:15:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:15:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:15:15] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:15:39] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:45] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:16:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:16:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:18:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82900 MB (13% inode=99%): [09:21:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 92668 MB (9% inode=99%): [09:21:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:24:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:25:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36284.000000 [09:26:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:27:49] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:29:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:30:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36560.000000 [10:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:15:04] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:15:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:15:15] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:15:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:15:46] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:16:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:16:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:18:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82631 MB (13% inode=99%): [10:21:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:23:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:24:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:25:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39885.000000 [10:26:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:27:46] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:29:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:30:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21730.000000 [10:47:45] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:47:55] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:47:55] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:49:07] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 63503 MB (6% inode=97%): [10:49:07] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30268 MB (7% inode=99%): [10:49:07] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 63503 MB (6% inode=97%): [10:54:15] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [10:54:25] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:06:37] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3183.000000 [11:09:39] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1579.000000 [11:11:15] MySQL on z-dat-s3-a is OK: Uptime: 1327 Threads: 16 Questions: 2063 Slow queries: 2 Opens: 349 Flush tables: 1 Open tables: 337 Queries per second avg: 1.554 [11:11:26] MySQL slave on z-dat-s3-a is OK: Uptime: 1334 Threads: 17 Questions: 2172 Slow queries: 2 Opens: 397 Flush tables: 1 Open tables: 385 Queries per second avg: 1.628 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1424 [11:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:15:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:15:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:15:16] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:15:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:46] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:17:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:17:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:19:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82389 MB (13% inode=99%): [11:23:01] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:23:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:25:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:25:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43485.000000 [11:26:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:27:46] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:29:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:38:39] hello all [11:45:22] hello [11:45:39] the toolserver will go offline for some minutes [11:46:20] we will exchange the broken switch [11:50:25] DaBPunkt: hello, has query-killer gone away again? [11:50:59] on rosemary, at least? [11:55:39] might be because rosemary was offline again [12:08:59] ethernet 0/1/11 [msw1-oe10-esams] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [12:14:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:15:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:15:16] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:15:16] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:15:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:15:45] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:16:59] ethernet 0/1/11 [msw1-oe10-esams] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/11:UP:1 UP: OK [12:18:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:18:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:18:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82128 MB (13% inode=99%): [12:22:57] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:23:46] CAM on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:46] switch of switches done [12:31:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47463.000000 [12:32:46] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 29525 MB (7% inode=99%): [12:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:32:46] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 64432 MB (6% inode=97%): [12:32:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82081 MB (13% inode=99%): [12:32:59] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 64432 MB (6% inode=97%): [12:32:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:33:05] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:33:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:33:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:33:35] aliasd on yarrow is CRITICAL: Connection refused [12:46:03] oh man..this aliasd... [13:08:35] aliasd on yarrow is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [13:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:15:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:15:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:15:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:16:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:18:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:18:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:22:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:27:08] anyone willing to run SQL select for me please? [13:31:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51063.000000 [13:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81803 MB (13% inode=99%): [13:32:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:33:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5216778 [13:33:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:34:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:15:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:15:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:15:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:16:18] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:18:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:18:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:31:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54663.000000 [14:32:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:32:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81558 MB (13% inode=99%): [14:32:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:33:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5203151 [14:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:34:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:39:50] @replag [14:39:50] DaBPunkt: s1-user: 15h 19m 18s [+1.00 s/s] [14:40:21] nosy: did you see "RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1"? [14:41:28] @replag [14:41:29] DaBPunkt: s1-user: 15h 20m 56s [+0.99 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s] [14:42:03] Was the switch maintenance done and successful? [14:42:40] [26.10.2012 13:45:39] the toolserver will go offline for some minutes [14:42:40] [26.10.2012 13:46:20] we will exchange the broken switch [14:42:41] I guess so [14:43:13] I wasn't online at that time [14:43:41] it's a bit annoying she didn't report it after asking explicitely :P [14:44:06] Platonides: yes [14:44:33] * DaBPunkt didn't knew that there was a broken switch at all [14:46:10] wow, that's probably more annoying :) [15:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:15:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:15:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:15:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:18:35] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:18:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:24:44] @replag [15:24:44] DaBPunkt: s1-user: 16h 4m 12s [+1.00 s/s]; s3-rr-a: 48s [+0.01 s/s]; s3-user: 48s [+0.01 s/s] [15:25:18] strange. I see a decreasing replag [15:32:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58324.000000 [15:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81283 MB (13% inode=99%): [15:33:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:33:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5192752 [15:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:34:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:01:45] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:01:46] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:01:55] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:04:49] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 30927 MB (7% inode=99%): [16:04:49] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 64071 MB (6% inode=97%): [16:04:55] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 64071 MB (6% inode=97%): [16:10:36] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [16:10:45] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [16:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:15:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:15:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:15:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:16:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:18:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:18:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:21:46] MySQL slave on z-dat-s3-a is OK: Uptime: 1048 Threads: 1 Questions: 6 Slow queries: 0 Opens: 20 Flush tables: 1 Open tables: 13 Queries per second avg: 0.5 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:22:36] MySQL on z-dat-s3-a is OK: Uptime: 1097 Threads: 9 Questions: 2974 Slow queries: 1 Opens: 258 Flush tables: 1 Open tables: 247 Queries per second avg: 2.711 [16:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:32:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61924.000000 [16:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:32:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81011 MB (13% inode=99%): [16:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:33:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5185496 [16:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:34:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:34:09] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:15:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:15:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:15:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:16:06] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:16:16] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:18:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:18:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:32:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65524.000000 [17:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:32:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80685 MB (13% inode=99%): [17:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:33:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5175448 [17:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:34:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:35:36] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:00:29] DaBPunkt: FYI, rosemary's stopped replicating [18:14:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:15:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:15:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:15:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:23:56] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:32:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69123.000000 [18:32:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80428 MB (13% inode=99%): [18:33:09] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5164183 [18:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:33:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:34:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:34:09] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:56:33] rosemary's sql-partition is full [18:58:45] looks like it is finaly broken [19:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:15:40] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:15:40] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:15:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:19:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:19:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:32:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:32:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80146 MB (13% inode=99%): [19:33:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72743.000000 [19:33:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:33:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:33:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:34:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:41:05] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [19:41:15] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:56:15] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1314.000000 [19:57:05] MySQL on rosemary is OK: Uptime: 1363 Threads: 21 Questions: 9363 Slow queries: 23 Opens: 232 Flush tables: 1 Open tables: 223 Queries per second avg: 6.869 [19:59:56] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199269 MB (20% inode=99%): [20:03:46] Seems rather news worthy [20:04:58] So are we going to drop commons off rosemary as well? [20:12:49] Dispenser: there SHOULD be enough space on rosemary for enwp+commons. For unknown reason mysql created 2 douzend relay-bin-files until no more space was left [20:14:03] somewhen tomorrow or sunday the wmf-dump will be tranfered so we can re-setup rosemary or thyme anyway [20:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:15:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:15:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:15:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:16:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:24:00] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79878 MB (13% inode=99%): [20:33:06] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:33:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76343.000000 [20:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:33:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:33:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:34:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:34:09] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:44:48] Platonides: well, they take some significant space and are old, can you consider packing? [21:11:54] Platonides: incr_Nov-Mar is 132,2 GB [21:14:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:15:05] MySQL slave on rosemary is WARNING: No slaves defined [21:15:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:15:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:15:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:16:17] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 135441 MB (13% inode=99%): [21:17:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106688 MB (10% inode=99%): [21:18:05] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:23:54] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79505 MB (13% inode=99%): [21:33:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79943.000000 [21:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:33:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:33:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:34:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:09:53] zzz [22:14:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:15:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:15:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:15:35] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:16:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:18:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5236656 [22:19:36] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:19:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:23:55] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79248 MB (13% inode=99%): [22:33:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83543.000000 [22:33:15] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:33:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:33:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:34:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:34:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:53:15] CAM on hemlock is OK: OK - cam detected no new errors [23:23:23] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:23:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:23:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:23:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:23:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:23:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5224210 [23:23:25] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:23:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:25:00] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:32:37] nacht ts [23:32:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79014 MB (12% inode=99%): [23:33:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87142.000000 [23:33:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:33:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:34:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:35:05] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1