[00:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:01:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:02:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:02:19] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:02:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:02:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:03:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18735 [00:04:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18734.000000 [00:09:58] /sql on cassia is WARNING: DISK WARNING - free space: /sql 116261 MB (9% inode=99%): [00:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95043 MB (15% inode=99%): [00:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:30:00] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70304 MB (5% inode=99%): [00:30:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:36:59] /sql on cassia is OK: DISK OK - free space: /sql 196965 MB (16% inode=99%): [01:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:01:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:02:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:02:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:02:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:02:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:03:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:03:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19821 [01:04:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19836.000000 [01:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94807 MB (15% inode=99%): [01:21:42] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:30:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:01:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:02:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:02:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:02:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:02:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:02:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:03:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19391 [02:04:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19366.000000 [02:07:17] Load avg. on ortelius is WARNING: WARNING - load average: 23.66, 21.85, 13.44 [02:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94593 MB (15% inode=99%): [02:21:19] Load avg. on ortelius is OK: OK - load average: 9.52, 13.96, 14.72 [02:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:29:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:44:17] Load avg. on ortelius is WARNING: WARNING - load average: 21.50, 17.64, 15.72 [02:53:08] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:57:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:01:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:02:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:02:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:02:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:02:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:03:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:03:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17881 [03:04:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17893.000000 [03:09:18] Load avg. on ortelius is OK: OK - load average: 7.43, 10.84, 14.59 [03:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94474 MB (15% inode=99%): [03:21:41] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:29:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:54:48] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 65710 MB (6% inode=97%): [03:54:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 65637 MB (6% inode=97%): [04:01:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:01:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:02:10] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:02:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:02:39] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:02:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:03:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:03:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16867 [04:04:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16871.000000 [04:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94354 MB (15% inode=99%): [04:21:41] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:30:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:01:39] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:01:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:02:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:02:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:02:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:03:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:03:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16490 [05:05:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16464.000000 [05:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94241 MB (15% inode=99%): [05:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:29:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:01:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:01:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:02:09] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:02:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:02:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:02:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:03:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:03:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16192 [06:05:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16222.000000 [06:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94097 MB (15% inode=99%): [06:21:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:30:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:31:18] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:41] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:41] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:01:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:02:01] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:02:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:02:46] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:02:46] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:03:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:03:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14748 [07:04:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14663.000000 [07:17:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93890 MB (15% inode=99%): [07:21:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:29:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:31:18] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:51] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:58:59] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200194 MB (20% inode=99%): [08:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:01:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:02:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:02:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:02:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:02:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:03:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11980 [08:04:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11915.000000 [08:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93701 MB (15% inode=99%): [08:21:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:29:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:31:18] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:48] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:32:40] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:52:28] aliasd on nightshade is CRITICAL: Connection refused [09:01:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:01:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:02:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:02:51] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:03:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:03:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8449 [09:05:18] Hi guys, do I need to file a Jira ticket to create a new project in Jira? [09:05:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8329.000000 [09:12:28] aliasd on nightshade is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [09:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93471 MB (15% inode=99%): [09:20:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 94811 MB (9% inode=99%): [09:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:29:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:31:18] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:32:19] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:32:27] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:32:27] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:32:38] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:33:57] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 66200 MB (6% inode=97%): [09:33:57] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 66200 MB (6% inode=97%): [09:33:58] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 32136 MB (7% inode=99%): [09:37:47] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:38:18] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:38:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:40:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:40:31] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:41:48] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:42:18] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:42:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:45:28] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:02:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:02:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:03:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:03:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3882 [10:05:21] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3783.000000 [10:07:17] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3569.000000 [10:07:48] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3496 [10:18:00] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93221 MB (15% inode=99%): [10:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:49] MySQL slave on rosemary is OK: Uptime: 141989 Threads: 9 Questions: 131627241 Slow queries: 37663 Opens: 12160 Flush tables: 1 Open tables: 2982 Queries per second avg: 927.24 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1768 [10:25:18] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1745.000000 [10:27:18] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:27:38] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:27:50] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:53:31] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 66071 MB (6% inode=97%): [11:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:02:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:02:40] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:03:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:03:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92934 MB (15% inode=99%): [11:18:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:02:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:02:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92673 MB (15% inode=99%): [12:18:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:19:38] Platonides: around? [12:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:02:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:02:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92435 MB (15% inode=99%): [13:18:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:22:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:31:22] hello all [13:42:20] Has anyone experienced mysql error 1028 using s1/enwiki_p before? [13:43:28] According to the mysql docs it may be due to insufficient disk space in /tmp, bad/not enough memory, query being killed, corruption, and a few other things. [13:43:34] Just wondering if I'm the only one [13:51:32] bjelleklang: which host do you use? s1-user or s1-rr? [14:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:02:21] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:02:37] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92124 MB (15% inode=99%): [14:18:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:22:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:24:48] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:29:27] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:58] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:48] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.034 second response time [14:37:22] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [15:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:02:00] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:02:47] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:02:57] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1921.000000 [15:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:03:48] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1927 [15:07:13] Marlen Caemmerer * [Toolserver-l] Maintenance s4-user [15:18:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91855 MB (15% inode=99%): [15:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:21:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:22:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:02:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:02:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:02:59] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2790.000000 [16:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:03:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:03:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:03:59] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2806 [16:14:07] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [16:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91599 MB (15% inode=99%): [16:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:22:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:38:57] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [16:40:58] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3208.000000 [16:44:07] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2167.000000 [16:44:08] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2172.000000 [16:44:48] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2208.000000 [16:48:28] aliasd on nightshade is CRITICAL: Connection refused [17:01:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:03:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:03:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:03:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:03:50] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3387 [17:08:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [17:08:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [17:09:16] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [17:13:17] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [17:13:51] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3361.000000 [17:14:08] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [17:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 91260 MB (15% inode=99%): [17:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:48:28] aliasd on nightshade is CRITICAL: Connection refused [18:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:02:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:03:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:03:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:03:48] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3062 [18:08:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7207.000000 [18:08:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7212.000000 [18:08:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7248.000000 [18:14:08] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [18:14:28] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3131.000000 [18:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90974 MB (14% inode=99%): [18:21:41] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:25:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:38:53] hello all [18:39:48] anyone there? [18:42:42] Hello Annabel [18:48:27] aliasd on nightshade is CRITICAL: Connection refused [18:58:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [18:58:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3641 [19:01:38] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:02:11] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:02:18] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:02:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:03:51] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:03:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:08:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10807.000000 [19:08:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10812.000000 [19:08:51] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10849.000000 [19:14:08] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [19:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90699 MB (14% inode=99%): [19:21:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:22:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:38:52] Anyone here? [19:39:18] Has anyone else had any 1028 errors from mysql on s1? (enwiki_p) [19:48:29] aliasd on nightshade is CRITICAL: Connection refused [19:58:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5272.000000 [19:58:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5279 [19:59:00] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199652 MB (20% inode=99%): [20:01:10] Luca Libonati * [Toolserver-l] dump of attribution information [20:01:39] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:01:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:03:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:03:50] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:03:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:08:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14406.000000 [20:08:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14412.000000 [20:08:10] John * Re: [Toolserver-l] dump of attribution information [20:08:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14448.000000 [20:14:10] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [20:18:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90394 MB (14% inode=99%): [20:19:57] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 57364 MB (5% inode=97%): [20:19:58] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 57304 MB (5% inode=97%): [20:21:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:47:58] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59691 MB (6% inode=97%): [20:47:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59684 MB (6% inode=97%): [20:48:27] aliasd on nightshade is CRITICAL: Connection refused [20:54:58] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58429 MB (5% inode=97%): [20:54:58] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58421 MB (5% inode=97%): [20:58:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6739.000000 [20:58:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6744 [21:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:02:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:02:17] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:02:32] aliasd on nightshade is OK: TCP OK - 0.004 second response time on port 984 [500 Not found.] [21:02:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:03:18] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:03:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:04:44] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:08:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16152.000000 [21:08:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17479.000000 [21:08:52] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13098.000000 [21:09:27] aliasd on nightshade is CRITICAL: Connection refused [21:12:09] Marlen Caemmerer * Re: [Toolserver-announce] Maintenance s4-user [21:14:10] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10869 [21:15:58] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59561 MB (6% inode=97%): [21:15:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59561 MB (6% inode=97%): [21:17:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90078 MB (14% inode=99%): [21:20:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 93310 MB (9% inode=99%): [21:21:41] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:25:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:31:48] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3184.000000 [21:32:07] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2965 [21:34:08] MySQL slave on daphne is OK: Uptime: 20374731 Threads: 19 Questions: 1689515314 Slow queries: 657662 Opens: 148852 Flush tables: 3 Open tables: 1162 Queries per second avg: 82.922 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1707 [21:34:52] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1421.000000 [21:34:52] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 31314 MB (7% inode=99%): [21:38:19] Hello all. I have a question: Is operating interwiki bot to run on a list of pages (across wikis) in screen mode, considered as "continuous running" banned by "No interwikibots" rule? [21:53:21] / on wolfsbane is WARNING: DISK WARNING - free space: / 6284 MB (20% inode=93%): [21:53:27] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6284 MB (20% inode=93%): [21:55:18] / on wolfsbane is OK: DISK OK - free space: / 6294 MB (21% inode=93%): [21:56:29] /tmp on wolfsbane is OK: DISK OK - free space: / 6297 MB (21% inode=93%): [21:58:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7597.000000 [21:58:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7608 [22:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:02:07] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:02:22] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:03:20] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:03:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:04:18] / on wolfsbane is WARNING: DISK WARNING - free space: / 6245 MB (20% inode=93%): [22:04:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:08:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5757.000000 [22:08:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13196.000000 [22:09:28] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6289 MB (20% inode=93%): [22:09:28] aliasd on nightshade is CRITICAL: Connection refused [22:17:08] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3430.000000 [22:17:09] DaB. * [Toolserver-l] Current progress [22:17:27] aliasd on nightshade is OK: TCP OK - 0.004 second response time on port 984 [500 Not found.] [22:18:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89803 MB (14% inode=99%): [22:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:22:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:28:08] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1719.000000 [22:38:02] nacht ts [22:43:28] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6262 MB (20% inode=93%): [22:58:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8618.000000 [22:58:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8628 [23:01:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:01:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:02:08] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:02:20] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:02:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:03:17] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:03:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:04:18] / on wolfsbane is WARNING: DISK WARNING - free space: / 6256 MB (20% inode=93%): [23:04:38] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:08:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4458.000000 [23:14:20] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3484.000000 [23:17:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89568 MB (14% inode=99%): [23:21:21] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1355.000000 [23:21:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:21:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:22:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:25:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:43:28] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6257 MB (20% inode=93%): [23:58:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7958.000000 [23:58:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7961