[00:16:24] Load avg. on willow is WARNING: WARNING - load average: 16.24, 16.22, 15.24 [00:19:23] Load avg. on willow is OK: OK - load average: 11.74, 14.32, 14.67 [00:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70347 MB (11% inode=99%): [00:23:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:24:54] nacht ts [00:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:36:24] Load avg. on willow is WARNING: WARNING - load average: 15.55, 15.37, 14.86 [00:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [00:43:24] Load avg. on willow is OK: OK - load average: 12.68, 14.68, 14.84 [01:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70205 MB (11% inode=99%): [01:23:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:29:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:34:53] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:38:31] @replag [01:38:32] Matthew_: s1-rr-a: 5m 44s [+0.10 s/s]; s1-user: 5m 44s [+0.10 s/s]; s2-user: 1d 9h 22m 27s [-2.54 s/s]; s3-rr-a: 44s [-0.00 s/s]; s3-user: 44s [-0.00 s/s] [01:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70081 MB (11% inode=99%): [02:23:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69981 MB (11% inode=99%): [03:23:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [04:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69854 MB (11% inode=99%): [04:23:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69642 MB (11% inode=99%): [05:23:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69502 MB (11% inode=99%): [06:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:49:00] can anybody tell me in short where tomcat writes the log files to? (at the toolserver setup) Somewhere at home I dropped a note about it, but would be useful now here [07:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69311 MB (11% inode=99%): [07:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69119 MB (11% inode=99%): [08:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:17:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 103833 MB (10% inode=99%): [09:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68983 MB (11% inode=99%): [09:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68795 MB (11% inode=99%): [10:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:56:24] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36939 MB (9% inode=99%): [10:56:34] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73081 MB (7% inode=98%): [10:56:34] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73081 MB (7% inode=98%): [11:18:06] DaB. * Re: [Toolserver-l] When to execute cron-tasks [11:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68618 MB (11% inode=99%): [11:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:47:39] hello all [12:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68458 MB (11% inode=99%): [12:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68291 MB (11% inode=99%): [13:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:25:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:04:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:19:33] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68095 MB (11% inode=99%): [14:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:25:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67933 MB (11% inode=99%): [15:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:25:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:23:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67680 MB (11% inode=99%): [16:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:25:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:40:23] wtf? i'm being spammed by cron notifications [16:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:42:35] Hi [16:43:46] iH [16:44:06] Dr. Trigon * Re: [Toolserver-l] When to execute cron-tasks [16:44:06] Danny B. * Re: [Toolserver-l] When to execute cron-tasks [16:44:24] Anyone here who could run me a quick sql query? need a list of all articles on deWiki and their current revision [16:44:44] format doesnt matter,as long as it's machine-readable [16:45:00] no prob [16:46:05] but it will take a while i guess [16:46:55] yeah, thats no problem [16:47:02] HardDisk_WP: redirects yes/no? [16:47:54] yeah, can you please make a column with the target article-id in case the article is a redirect? [16:48:06] Danny_B|backup: the mail problem is unrelated. there was a problem on nightshade mail service and you got all mails queued since about two weeks [16:49:13] HardDisk_WP: eh, i can give you 1/0 if it's redir or not [16:49:58] hmm ok, that's fine too [16:50:48] running [16:51:19] cool, thanks :) [16:52:23] it will be machine readable, but you'll have to trim extra spaces [16:53:08] hmm, actually you can get this from dump as well, but it will be accurate to the dump date [16:53:42] the redirect-target? [16:54:05] both revisions and target [16:54:16] but anyway, i can give you targets in another table [16:54:38] can you make it consistent with the first dump? [16:58:49] you'll see [16:59:43] cool :) [17:01:06] DaB. * [Toolserver-l] Possible mail-flood [17:14:54] DaBPunkt, ping [17:15:08] omg how many redirs you have, the query is still writing them to output [17:16:08] hey we have redirects, enWP has articles about every kind of junk on this planet [17:16:20] and stub templates for every category of junk :D [17:17:52] DaBPunkt, uh... can you please revive my old TS account? I don't have anything anymore, neither the ssh-key nor any passwords and i don't have any idea which email-address the account has [17:18:27] annihilate all stub templates! [17:18:34] they are worth nothing [17:18:59] HardDisk_WP: DaBPunkt ist busy, open a jira ticket [17:19:56] oh noes, not another site without SSO :( [17:20:53] HardDisk_WP: got sed/awk? [17:21:01] yep, on my server [17:21:23] oki, so i'll let the formatting on you [17:22:49] wtf thats fast [17:23:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67527 MB (11% inode=99%): [17:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:25:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:25:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:25:45] network fail?^^ [17:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:38:22] HardDisk_WP: 1020639 rows in set (redirs) [17:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:47:20] Danny_B|backup, holy crap :D [17:47:48] Danny_B|backup, can you please put both resultsets on somewhere I can download them via wget? [17:47:56] hmm, some redirs go from main ns to different ns, shit [17:48:31] Danny_B|backup, thats no problem [17:48:45] it is because i did not output target namespace [17:49:00] though it'd be interesting which ones do that. AFAIR the only page allowed to do this is the mainpage [17:52:12] there is obviously more, at least Wikiquette [18:00:23] Free Memory on damiana is WARNING: WARNING - 6.1% (510556 kB) free! [18:02:24] Free Memory on damiana is OK: OK - 7.3% (608460 kB) free. [18:23:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67359 MB (11% inode=99%): [18:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:25:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:57:00] 1h left before the arch switch [19:00:06] DaB. * Re: [Toolserver-l] When to execute cron-tasks [19:20:34] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1858 [19:21:03] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1854.000000 [19:22:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67083 MB (10% inode=99%): [19:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:25:04] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1788.000000 [19:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:25:34] MySQL slave on rosemary is OK: Uptime: 15566888 Threads: 12 Questions: 6908283071 Slow queries: 2276105 Opens: 332890 Flush tables: 6 Open tables: 4154 Queries per second avg: 443.780 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1773 [19:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:38:33] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3941 MB (17% inode=99%): [19:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:52:04] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1926.000000 [19:52:33] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1944 [20:22:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66790 MB (10% inode=99%): [20:23:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:25:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:38:33] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3979 MB (17% inode=99%): [20:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:52:03] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2952.000000 [20:52:33] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2966 [20:52:33] /tmp on ortelius is OK: DISK OK - free space: /tmp 14233 MB (61% inode=99%): [21:03:23] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [21:03:24] Sun Grid Engine execd on nightshade is UNKNOWN: Error with qhost: error: commlib error: got select error (Connection refused) [21:17:23] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [21:17:23] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [21:17:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 102227 MB (10% inode=99%): [21:22:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66453 MB (10% inode=99%): [21:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:25:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:29:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3613 [21:30:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [21:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66221 MB (10% inode=99%): [22:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:54] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:26:05] DaB. * Re: [Toolserver-announce] [Toolserver-l] Switching of SGE-arch-default at 9. October [22:28:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:29:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3721 [22:30:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3737.000000 [22:32:24] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:35:33] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3579 [22:36:04] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3568.000000 [22:42:03] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:56:23] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36972 MB (9% inode=99%): [22:56:33] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 72669 MB (7% inode=98%): [22:56:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 72669 MB (7% inode=98%): [23:15:04] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1761.000000 [23:15:33] MySQL slave on rosemary is OK: Uptime: 15580689 Threads: 22 Questions: 6915210579 Slow queries: 2281497 Opens: 333691 Flush tables: 6 Open tables: 4166 Queries per second avg: 443.832 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1752 [23:22:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66027 MB (10% inode=99%): [23:23:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:49] @replag [23:24:50] DaBPunkt: s1-rr-a: 8m 37s [+0.01 s/s]; s1-user: 8m 37s [+0.01 s/s]; s2-user: 27s [-0.41 s/s]; s3-rr-a: 26s [+0.00 s/s]; s3-user: 26s [+0.00 s/s] [23:24:53] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:25:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:25:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:25:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:28:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:32:23] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:39:36] nacht ts [23:42:04] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: