[00:14:04] nacht ts [00:38:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:42:35] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:42:35] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [00:42:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:45:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:46:41] -18 [00:47:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:51:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:51:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:51:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:53:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:53:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:57:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:38:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:42:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:42:07] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [01:43:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:45:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:47:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:51:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:51:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:51:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:53:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:53:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:57:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:38:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:43:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [02:43:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:45:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:47:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:52:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:52:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:53:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:53:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:57:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:21:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:39:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:43:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [03:43:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:46:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:47:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:52:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:52:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:54:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:54:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:57:33] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:21:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:39:09] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 60186 MB (6% inode=97%): [04:39:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:40:09] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 60194 MB (6% inode=97%): [04:43:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:43:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [04:43:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:46:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:46:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:47:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:52:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:52:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:54:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:54:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:57:36] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:39:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:43:08] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:43:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [05:43:36] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:46:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:46:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:48:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:52:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:52:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:54:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:54:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:57:36] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:39:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:43:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:43:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [06:44:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:46:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:48:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:51:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:52:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:52:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:54:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:54:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:58:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:12:43] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 127915 MB (13% inode=99%): [07:39:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:44:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [07:44:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:46:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:46:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:48:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:52:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:53:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:54:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:54:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:58:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:02:42] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:03:15] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197560 MB (20% inode=99%): [08:04:04] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:04:05] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:04:43] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 28708 MB (7% inode=99%): [08:05:06] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 60121 MB (6% inode=97%): [08:05:06] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 60121 MB (6% inode=97%): [08:05:42] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:06:05] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:06:05] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:20:43] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:20:43] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [08:21:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:30:42] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 29323 MB (7% inode=99%): [08:31:05] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 60017 MB (6% inode=97%): [08:31:05] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 60017 MB (6% inode=97%): [08:32:44] MySQL on z-dat-s3-a is OK: Uptime: 1087 Threads: 6 Questions: 366 Slow queries: 1 Opens: 170 Flush tables: 1 Open tables: 162 Queries per second avg: 0.336 [08:32:44] MySQL slave on z-dat-s3-a is OK: Uptime: 1088 Threads: 6 Questions: 389 Slow queries: 1 Opens: 174 Flush tables: 1 Open tables: 166 Queries per second avg: 0.357 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1089 [08:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:44:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:45:08] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:45:08] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [08:47:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:47:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:48:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:52:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:54:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:54:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:54:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:58:39] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:04:05] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2214 MB (18% inode=99%): [09:40:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:44:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:45:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [09:47:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:47:06] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:48:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:51:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:53:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:54:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:54:30] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:58:46] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:01:10] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 751 MB (6% inode=99%): [10:02:04] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2003 MB (16% inode=99%): [10:03:05] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 351 MB (2% inode=99%): [10:41:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:44:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:46:09] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:09] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [10:47:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:51:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:53:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:55:24] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:55:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:55:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:59:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:03:05] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 146 MB (1% inode=99%): [11:05:27] Sun Grid Engine execd on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:06:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:08:13] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:08:13] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:11] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 144 MB (1% inode=99%): [11:18:05] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.006 second response time [11:18:16] /tmp on ortelius is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:34:07] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58016 MB (5% inode=97%): [11:34:07] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 57982 MB (5% inode=97%): [11:39:05] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59747 MB (6% inode=97%): [11:39:05] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59747 MB (6% inode=97%): [11:39:07] DaB. * Re: [Toolserver-l] Welcome: wikidata [11:40:07] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1631 MB (12% inode=99%): [11:41:45] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:41:51] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:42:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:04] /tmp on ortelius is OK: DISK OK - free space: /tmp 10818 MB (81% inode=99%): [11:45:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:45:43] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [11:47:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:49:54] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:49:54] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:51:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:53:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:54:04] /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58310 MB (5% inode=97%): [11:54:04] /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58342 MB (5% inode=97%): [11:56:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:57:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58932 MB (6% inode=97%): [11:57:04] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58970 MB (6% inode=97%): [12:00:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:05:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:09:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:45:35] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:46:04] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [12:47:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:49:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:51:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:51:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:54:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:56:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:00:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:05:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:09:12] @replag [13:09:12] DaBPunkt: s1-rr-a-wd: 21h 25m 43s [+0.97 s/s]; s1-user-wd: 21h 59m 22s [+1.00 s/s]; s2-user-wd: 2d 18h 40m 19s [+1.00 s/s]; s3-rr-a: 2m 40s [+0.00 s/s]; s3-user: 2m 40s [+0.00 s/s]; s4-user-wd: 2d 17h 37m 10s [+1.00 s/s]; s5-user-wd: 2d 18h 40m 19s [+1.00 s/s] [13:09:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:45:39] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:46:04] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:46:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [13:47:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:48:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:49:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:52:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:54:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:56:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:00:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:05:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:10:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:21:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:44:01] @replag [14:44:15] DaBPunkt: s1-rr-a-wd: 22h 58m 57s [+0.98 s/s]; s1-user-wd: 23h 34m 18s [+1.00 s/s]; s2-user-wd: 2d 20h 15m 15s [+1.00 s/s]; s3-rr-a: 36s [-0.02 s/s]; s3-user: 36s [-0.02 s/s]; s4-user-wd: 2d 19h 12m 6s [+1.00 s/s]; s5-user-wd: 2d 20h 15m 15s [+1.00 s/s] [14:46:12] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:46:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [14:46:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:47:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:48:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:53:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:55:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:57:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:58:15] @replag [14:58:15] DaBPunkt: s1-rr-a-wd: 23h 9m 59s [+0.79 s/s]; s1-user: 16s [-0.00 s/s]; s1-user-wd: 23h 48m 20s [+1.00 s/s]; s2-user: 11s [-0.00 s/s]; s2-user-c: 24s [-0.00 s/s]; s2-user-wd: 2d 20h 29m 17s [+1.00 s/s]; s3-rr-a: 59s [+0.03 s/s]; s3-user: 59s [+0.03 s/s] [14:58:15] DaBPunkt: s4-user-wd: 2d 19h 26m 8s [+1.00 s/s]; s5-user-c: 24s [-0.00 s/s]; s5-user-wd: 2d 20h 29m 17s [+1.00 s/s] [14:58:21] @replag [14:58:22] DaBPunkt: s1-rr-a-wd: 23h 9m 47s [-1.07 s/s]; s1-user-wd: 23h 48m 31s [+0.98 s/s]; s2-user-c: 31s [+0.62 s/s]; s2-user-wd: 2d 20h 29m 28s [+0.98 s/s]; s3-rr-a: 1m 10s [+0.98 s/s]; s3-user: 1m 10s [+0.98 s/s]; s4-user-wd: 2d 19h 26m 19s [+0.98 s/s]; s5-user-c: 31s [+0.62 s/s] [14:58:23] DaBPunkt: s5-user-wd: 2d 20h 29m 28s [+0.98 s/s] [14:59:55] @replag [14:59:55] DaBPunkt: s1-rr-a-wd: 23h 7m 4s [-1.84 s/s]; s1-user-wd: 23h 50m 0s [+1.01 s/s]; s2-user: 10s [-0.01 s/s]; s2-user-wd: 2d 20h 30m 57s [+1.01 s/s]; s3-rr-a: 2m 39s [+1.01 s/s]; s3-user: 2m 39s [+1.01 s/s]; s4-user-wd: 2d 19h 27m 48s [+1.01 s/s]; s5-user-wd: 2d 20h 30m 57s [+1.01 s/s] [15:01:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:04:46] @replag [15:04:47] DaBPunkt: s1-rr-a-wd: 22h 53m 47s [-2.69 s/s]; s1-user-wd: 23h 54m 56s [+1.00 s/s]; s2-user: 11s [+0.00 s/s]; s2-user-wd: 2d 20h 35m 53s [+1.00 s/s]; s4-user-wd: 2d 19h 32m 44s [+1.00 s/s]; s5-user-wd: 2d 20h 35m 53s [+1.00 s/s]; s7-rr-a: 10s [-0.00 s/s]; s7-user: 10s [-0.00 s/s] [15:06:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:10:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:20:33] @replag [15:20:34] DaBPunkt: s1-rr-a-wd: 23h 56m 29s [+3.97 s/s]; s1-user-wd: 1d 10m 43s [+1.00 s/s]; s2-user-wd: 2d 20h 51m 41s [+1.00 s/s]; s3-rr-a: 23s [-0.11 s/s]; s3-user: 23s [-0.11 s/s]; s4-user-wd: 2d 19h 48m 32s [+1.00 s/s]; s5-user-wd: 2d 20h 51m 41s [+1.00 s/s] [15:21:19] DaBPunkt: hello, is there an estimated time for fixing the broken replication on s1-user? [15:21:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:22:26] @replag [15:22:27] DaBPunkt: s1-rr-a-wd: 23h 49m 54s [-3.50 s/s]; s1-user-wd: 1d 12m 36s [+1.00 s/s]; s2-user-wd: 2d 20h 53m 33s [+0.99 s/s]; s3-rr-a: 1m 4s [+0.36 s/s]; s3-user: 1m 4s [+0.36 s/s]; s4-user-wd: 2d 19h 50m 24s [+0.99 s/s]; s5-user-wd: 2d 20h 53m 33s [+0.99 s/s]; s7-rr-a: 10s [-] [15:22:28] DaBPunkt: s7-user: 10s [-] [15:23:12] RussBlau: by "replication" you mean a.) the replication or b.) the broken data? [15:24:34] DaBPunkt: well, status.toolserver.org says "sql-s1-user Replication broken", so whatever that means, that is what I am asking about [15:27:04] it means that nosy did not updated status.toolserver.org after she repaired the replication-problem. Wait a moment [15:27:41] fixed [15:30:39] @replag [15:30:39] DaBPunkt: s1-rr-a-wd: 23h 30m 46s [-2.33 s/s]; s1-user-wd: 1d 20m 49s [+1.00 s/s]; s2-user-wd: 2d 21h 1m 46s [+1.00 s/s]; s3-rr-a: 15s [-0.10 s/s]; s3-user: 15s [-0.10 s/s]; s4-user-wd: 2d 19h 58m 37s [+1.00 s/s]; s5-user-wd: 2d 21h 1m 46s [+1.00 s/s] [15:37:37] DabPunkt: but the database (enwiki_p) is still corrupt, right? [15:37:54] yes, we work on that [15:38:32] DaBPunkt: thanks, any idea how long it will take? [15:38:45] a week [15:39:20] DaBPunkt: ok (sigh), thanks [15:46:37] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:47:10] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:10] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [15:48:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:49:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:53:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:55:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:58:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:01:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:06:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:10:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:21:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:43:08] @replag [16:43:08] DaBPunkt: s1-rr-a-wd: 1d 26m 42s [+0.77 s/s]; s1-user-wd: 1d 1h 33m 17s [+1.00 s/s]; s2-user: 11s [-]; s2-user-wd: 2d 22h 14m 14s [+1.00 s/s]; s4-user-wd: 2d 21h 11m 5s [+1.00 s/s]; s5-user-wd: 2d 22h 14m 14s [+1.00 s/s] [16:43:57] @replag [16:43:57] DaBPunkt: s1-rr-a-wd: 1d 25m 38s [-1.47 s/s]; s1-user-wd: 1d 1h 34m 1s [+1.01 s/s]; s2-user-wd: 2d 22h 14m 58s [+1.01 s/s]; s3-rr-a: 49s [+0.01 s/s]; s3-user: 49s [+0.01 s/s]; s4-user-wd: 2d 21h 11m 49s [+1.01 s/s]; s5-user-wd: 2d 22h 14m 58s [+1.01 s/s] [16:44:46] away for ma,ing dinner [16:47:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:47:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [16:47:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:48:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:49:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:50:52] hi there, anyone knows where is the description for user_properties_anonym table? [16:54:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:55:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:59:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:01:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:06:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:10:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:21:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:48:06] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:48:06] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [17:48:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:48:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:50:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:55:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:56:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:59:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:02:33] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:06:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:10:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:21:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:34:59] @replag [18:35:00] DaBPunkt: s1-rr-a-wd: 23h 27m 6s [-0.53 s/s]; s1-user-wd: 1d 3h 25m 9s [+1.00 s/s]; s2-user-wd: 3d 6m 6s [+1.00 s/s]; s3-rr-a: 2m 42s [+0.02 s/s]; s3-user: 2m 42s [+0.02 s/s]; s4-user-wd: 2d 23h 2m 57s [+1.00 s/s]; s5-user-wd: 3d 6m 6s [+1.00 s/s] [18:39:05] Hello [18:39:13] Anyone can help me?? [18:48:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:48:39] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:49:11] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:11] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [18:50:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:53:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:53:23] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [18:53:24] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [18:55:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:56:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:59:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:00:24] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1027.000000 [19:00:24] MySQL on daphne is OK: Uptime: 789 Threads: 4 Questions: 75770 Slow queries: 3 Opens: 78 Flush tables: 1 Open tables: 63 Queries per second avg: 96.32 [19:02:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:06:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:10:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:12:43] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 127739 MB (13% inode=99%): [19:17:54] Anyone can help me??\ [19:18:41] shuaib_: what's your problem? [19:19:14] @replag [19:19:14] DaBPunkt: s1-rr-a-wd: 22h 55m 42s [-0.71 s/s]; s1-user-wd: 1d 4h 9m 20s [+1.00 s/s]; s2-user-wd: 3d 50m 17s [+1.00 s/s]; s4-rr-a: 36m 4s [+0.01 s/s]; s4-user: 36m 4s [+0.01 s/s]; s4-user-wd: 2d 22h 39m 24s [-0.53 s/s]; s5-user-wd: 3d 50m 17s [+1.00 s/s] [19:19:15] I requsted for creating account on toolserver. but reply did nt come yet. [19:19:23] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2167.000000 [19:21:13] shuaib_: which number? [19:21:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:21:42] dont remember :( [19:28:26] shuaib_: https://jira.toolserver.org/browse/ACCAPP-571 maybe? [19:30:04] yes yes. [19:31:40] will approve or not? [19:32:12] It is aproved since 14. October. [19:32:46] it waits that you insert your information [19:33:40] which information? and where i should insert them? [19:36:01] shuaib_: There is a comment by me on this page. Do you see that? (maybe you have to scrool down) [19:36:21] yes I see. [19:36:36] but cannot find insert information link. [19:37:10] that should be at the top of the page [19:37:39] Do you see the button called "Comment"? [19:39:34] yes brother. [19:39:55] should I click it? [19:40:40] if you can not find the "insert information"-link which should be near, you can use the "comment"-button too [19:41:04] okeh I will comment now. [19:43:28] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [19:48:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:48:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:49:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:49:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [19:50:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:54:23] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [19:55:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:56:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:59:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:02:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:03:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197237 MB (20% inode=99%): [20:07:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:10:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:26:34] @replag [20:26:35] DaBPunkt: s1-rr-a-wd: 19h 20m 20s [-3.19 s/s]; s1-user-wd: 1d 5h 16m 44s [+1.00 s/s]; s2-user-wd: 3d 1h 57m 41s [+1.00 s/s]; s3-rr-a: 2m 44s [+0.00 s/s]; s3-user: 2m 44s [+0.00 s/s]; s4-rr-a: 1h 43m 28s [+1.00 s/s]; s4-user: 1h 43m 28s [+1.00 s/s]; s4-user-wd: 2d 23h 46m 48s [+1.00 s/s] [20:26:36] DaBPunkt: s5-user-wd: 3d 1h 57m 41s [+1.00 s/s] [20:30:44] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 28888 MB (7% inode=99%): [20:43:38] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7217.000000 [20:46:39] DaBPunkt_: I have inserted information as comment on that page. [20:48:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:49:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:49:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [20:49:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:51:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:51:33] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:54:23] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [20:54:54] shuaib_: thanks. I will look and handle it at sunday [20:55:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:55:55] okeh, I will wait. how to khow that account is to be created? [20:56:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:59:27] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:02:33] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:06:44] shuaib_: you will get an email [21:07:01] okeh and many thanks :) [21:07:06] np [21:07:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:11:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:44] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10827.000000 [21:48:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:49:04] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:04] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:49:05] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [21:49:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:51:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:51:35] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:54:27] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [21:55:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:56:36] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:56:52] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:57:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:00:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:02:38] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:07:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:11:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:25:54] @replag [22:25:54] DaBPunkt: s1-rr-a-wd: 1d 4h 36m 16s [+4.66 s/s]; s1-user-wd: 1d 4h 58m 48s [-0.15 s/s]; s2-user-wd: 3d 3h 57m 1s [+1.00 s/s]; s3-rr-a: 57s [-0.01 s/s]; s3-user: 57s [-0.01 s/s]; s4-rr-a: 3h 42m 48s [+1.00 s/s]; s4-user: 3h 42m 48s [+1.00 s/s]; s4-user-wd: 3d 1h 46m 8s [+1.00 s/s] [22:25:55] DaBPunkt: s5-user-wd: 3d 3h 57m 1s [+1.00 s/s] [22:28:55] @replag [22:28:55] DaBPunkt: s1-rr-a-wd: 1d 4h 17m 9s [-6.34 s/s]; s1-user-wd: 1d 4h 39m 55s [-6.26 s/s]; s2-user: 1m 22s [+0.00 s/s]; s2-user-c: 54s [+0.00 s/s]; s2-user-wd: 3d 4h 2s [+1.00 s/s]; s3-rr-a: 28s [-0.16 s/s]; s3-user: 28s [-0.16 s/s]; s4-rr-a: 3h 42m 19s [-0.16 s/s] [22:28:56] DaBPunkt: s4-user: 3h 42m 19s [-0.16 s/s]; s4-user-wd: 3d 1h 49m 9s [+1.00 s/s]; s5-user-c: 54s [+0.00 s/s]; s5-user-wd: 3d 4h 2s [+1.00 s/s] [22:39:28] @replag [22:39:29] DaBPunkt: s1-rr-a-wd: 1d 3h 43m 34s [-3.18 s/s]; s1-user-wd: 1d 4h 15s [-3.76 s/s]; s2-user-wd: 3d 4h 10m 35s [+1.00 s/s]; s3-rr-a: 11s [-0.03 s/s]; s3-user: 11s [-0.03 s/s]; s4-rr-a: 2h 48m 30s [-5.10 s/s]; s4-user: 2h 48m 30s [-5.10 s/s]; s4-user-wd: 3d 1h 59m 42s [+1.00 s/s] [22:39:30] DaBPunkt: s5-user-wd: 3d 4h 10m 35s [+1.00 s/s] [22:44:44] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8403.000000 [22:47:51] @replag [22:47:52] DaBPunkt: s1-rr-a-wd: 1d 3h 9m 56s [-4.01 s/s]; s1-user-wd: 1d 3h 33m 27s [-3.20 s/s]; s2-user-wd: 3d 4h 18m 58s [+1.00 s/s]; s3-rr-a: 1m 11s [+0.12 s/s]; s3-user: 1m 11s [+0.12 s/s]; s4-rr-a: 1h 59m 12s [-5.88 s/s]; s4-user: 1h 59m 12s [-5.88 s/s]; s4-user-wd: 3d 2h 8m 5s [+1.00 s/s] [22:47:53] DaBPunkt: s5-user-wd: 3d 4h 18m 58s [+1.00 s/s] [22:48:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:49:05] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:49:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [22:49:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:51:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:51:34] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:54:22] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5784 [22:57:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:57:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:00:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:01:23] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3402 [23:01:44] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3310.000000 [23:02:35] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:05:49] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1764.000000 [23:06:22] MySQL slave on daphne is OK: Uptime: 15548 Threads: 11 Questions: 4704683 Slow queries: 1108 Opens: 956 Flush tables: 1 Open tables: 215 Queries per second avg: 302.590 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1500 [23:07:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:11:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:11:51] @replag [23:11:51] DaBPunkt: s1-rr-a-wd: 1d 1h 44m 42s [-3.55 s/s]; s1-user-wd: 1d 2h 8m 9s [-3.55 s/s]; s2-user-wd: 3d 4h 42m 58s [+1.00 s/s]; s3-rr-a: 54s [-0.01 s/s]; s3-user: 54s [-0.01 s/s]; s4-user-wd: 3d 2h 32m 5s [+1.00 s/s]; s5-user-wd: 3d 4h 42m 58s [+1.00 s/s] [23:48:38] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:49:12] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21: [23:49:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:50:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:50:07] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:51:36] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:53:34] @replag [23:53:34] DaBPunkt: s1-rr-a: 20s [+0.00 s/s]; s1-rr-a-wd: 22h 43m 21s [-4.35 s/s]; s1-user: 25s [+0.00 s/s]; s1-user-wd: 23h 22m 10s [-3.99 s/s]; s2-user: 28s [-0.01 s/s]; s2-user-wd: 3d 5h 24m 37s [+1.00 s/s]; s3-rr-a: 57s [+0.00 s/s]; s3-user: 57s [+0.00 s/s] [23:53:34] DaBPunkt: s4-user-wd: 3d 3h 13m 44s [+1.00 s/s]; s5-user-wd: 3d 5h 24m 37s [+1.00 s/s] [23:57:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 60121 MB (6% inode=97%): [23:57:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:57:34] on myswl, how can i output to a .txt file? [23:57:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:58:05] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 60122 MB (6% inode=97%): [23:59:49] Alchimista: mysql -h sql-s1 -B -e "select max(rev_id) from recentchanges;">test.txt