[00:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:27:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:27:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30996 MB (5% inode=99%): [00:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:32:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:32:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:37:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [00:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 532663 MB (10% inode=44%): [00:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:54:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:57:43] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 115069 MB (11% inode=99%): [00:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:28:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:28:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30949 MB (5% inode=99%): [01:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:32:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:32:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:41:53] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [01:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 532560 MB (10% inode=44%): [01:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:54:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:27:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:28:45] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:28:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30753 MB (5% inode=99%): [02:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:32:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:32:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:37:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:42:53] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [02:45:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 530391 MB (10% inode=44%): [02:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:54:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:58:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:26:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:28:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:28:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30532 MB (4% inode=99%): [03:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:32:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:32:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:42:53] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [03:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 532286 MB (10% inode=44%): [03:47:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:54:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:59:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:29:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:29:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30431 MB (4% inode=99%): [04:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:32:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:32:24] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:43:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [04:45:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 532104 MB (10% inode=44%): [04:47:16] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:54:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:59:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:29:45] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:29:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30263 MB (4% inode=99%): [05:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:33:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:33:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:37:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:43:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [05:45:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531951 MB (10% inode=44%): [05:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:54:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:56:42] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 11068 MB (0% inode=94%): [05:59:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:27:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:30:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:30:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:30:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30104 MB (4% inode=99%): [06:33:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:34:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:37:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:43:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:45:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531735 MB (10% inode=44%): [06:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:54:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:59:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:27:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:31:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:31:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30001 MB (4% inode=99%): [07:33:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:34:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [07:45:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531595 MB (10% inode=44%): [07:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:54:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:59:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:01:10] test [08:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:28:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:31:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:31:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29901 MB (4% inode=99%): [08:33:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:34:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:45:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531381 MB (10% inode=44%): [08:45:52] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54238 [08:47:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:55:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:59:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:28:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:29:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 84812 MB (8% inode=98%): [09:29:43] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42050 MB (10% inode=99%): [09:29:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 84812 MB (8% inode=98%): [09:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:31:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:31:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29787 MB (4% inode=99%): [09:33:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:34:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:45:53] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20001 [09:46:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 531001 MB (10% inode=44%): [09:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:56:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:00:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:03:04] Marlen Caemmerer * Re: [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot of hyacinth s3/s6/s7) [10:14:52] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3355 [10:17:52] MySQL slave on z-dat-s3-a is OK: Uptime: 84347 Threads: 16 Questions: 53802820 Slow queries: 5887 Opens: 1152311 Flush tables: 1 Open tables: 16371 Queries per second avg: 637.874 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1774 [10:27:04] DaB. * [Toolserver-announce] Fwd: Re: [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot of hyacinth s3/s6/s7) [10:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:28:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:32:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29691 MB (4% inode=99%): [10:32:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:34:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:34:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:46:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 530678 MB (10% inode=44%): [10:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:56:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:01:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:29:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:32:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29562 MB (4% inode=99%): [11:32:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:35:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:38:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:47:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 530387 MB (10% inode=44%): [11:48:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:56:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:02:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:29:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:30:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:33:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:33:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29459 MB (4% inode=99%): [12:35:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:35:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:39:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:29:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:33:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:33:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29378 MB (4% inode=99%): [13:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:35:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:39:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:47:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 529904 MB (10% inode=44%): [13:48:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:57:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:02:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:08:04] Hersfold Wikipedia * Re: [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot [14:23:52] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:34:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:34:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29264 MB (4% inode=99%): [14:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:36:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:40:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:48:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:48:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 529675 MB (10% inode=44%): [14:57:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:03:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:27:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:30:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:34:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:34:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29185 MB (4% inode=99%): [15:35:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:36:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:48:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:48:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 529426 MB (10% inode=44%): [15:57:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:03:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:28:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:30:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:34:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:35:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29065 MB (4% inode=99%): [16:36:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:48:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 529155 MB (10% inode=44%): [16:49:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:58:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:03:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:29:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:34:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:36:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:36:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28977 MB (4% inode=99%): [17:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:49:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:49:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 528956 MB (9% inode=44%): [17:56:44] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 14407 MB (1% inode=95%): [17:58:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:04:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:29:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:34:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:36:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:37:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28878 MB (4% inode=99%): [18:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:49:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:49:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 528538 MB (9% inode=44%): [18:58:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:04:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:08:28] hello all [19:29:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:34:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:36:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28774 MB (4% inode=99%): [19:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:49:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:50:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 528127 MB (9% inode=44%): [19:58:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:04:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:29:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:34:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:36:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:38:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28637 MB (4% inode=99%): [20:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:49:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:50:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 527843 MB (9% inode=44%): [20:59:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:00:11] nacht ts [21:04:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:29:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:29:44] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42262 MB (10% inode=99%): [21:29:44] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 83916 MB (8% inode=98%): [21:29:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 83914 MB (8% inode=98%): [21:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:30:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:34:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:35:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:36:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:38:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28513 MB (4% inode=99%): [21:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:50:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 527605 MB (9% inode=44%): [22:00:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:04:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:29:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:30:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:34:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:38:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28391 MB (4% inode=99%): [22:40:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:50:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 527419 MB (9% inode=44%): [23:00:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:04:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:30:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:30:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:30:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:35:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:35:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:37:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:38:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28281 MB (4% inode=99%): [23:40:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:50:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:50:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 526056 MB (9% inode=44%):