[00:09:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:10:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:36:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:40:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:42:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:53:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:59:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 507672 MB (9% inode=43%): [00:59:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 114556 MB (11% inode=99%): [01:09:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:11:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:35:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:36:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:40:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:53:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:00:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 507617 MB (9% inode=43%): [02:09:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:11:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:37:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:54:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:00:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505441 MB (9% inode=43%): [03:10:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:12:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:38:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:42:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:54:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:00:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505474 MB (9% inode=43%): [04:10:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:13:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:38:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:55:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:59:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:01:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505366 MB (9% inode=43%): [05:10:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:13:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:03] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:38:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:41:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:55:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:58:53] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 7574 MB (0% inode=92%): [06:02:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505279 MB (9% inode=43%): [06:10:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:13:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:38:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:44:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:55:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:02:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505083 MB (9% inode=43%): [07:06:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 84151 MB (8% inode=98%): [07:06:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 84151 MB (8% inode=98%): [07:10:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:14:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:38:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:55:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:02:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 504844 MB (9% inode=43%): [08:10:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:11:53] Load avg. on adenia is WARNING: WARNING - load average: 17.55, 16.13, 12.83 [08:14:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:20:54] Load avg. on adenia is OK: OK - load average: 12.08, 14.63, 13.84 [08:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:39:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:42:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:45:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:55:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:03:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 504643 MB (9% inode=43%): [09:10:54] Load avg. on adenia is WARNING: WARNING - load average: 16.02, 13.57, 8.68 [09:11:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:13:53] Load avg. on adenia is OK: OK - load average: 13.91, 14.12, 9.81 [09:15:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42267 MB (10% inode=99%): [09:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:39:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:42:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:45:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:55:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:04:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 503991 MB (9% inode=43%): [10:12:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:16:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:42:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:46:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:56:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:04:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 503776 MB (9% inode=43%): [11:12:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:13:27] hello all [11:16:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:37:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:46:54] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:56:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:04:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 503469 MB (9% inode=43%): [12:12:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:16:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:42:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:46:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:56:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:59:52] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 115032 MB (11% inode=99%): [13:05:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506601 MB (9% inode=43%): [13:12:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:16:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:46:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:56:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:05:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505955 MB (9% inode=43%): [14:12:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:16:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:40:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:46:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:57:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:05:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505653 MB (9% inode=43%): [15:13:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:17:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:37:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:41:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:46:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:57:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:05:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505277 MB (9% inode=43%): [16:14:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:17:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:42:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:47:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:57:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:06:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 504722 MB (9% inode=43%): [17:14:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:17:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:42:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:43:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:47:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:57:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:58:53] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 9598 MB (0% inode=93%): [18:06:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 503007 MB (9% inode=43%): [18:15:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:18:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:36:21] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:43:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:47:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:57:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:06:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 501272 MB (9% inode=43%): [19:07:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 83778 MB (8% inode=98%): [19:07:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 83778 MB (8% inode=98%): [19:16:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:18:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:43:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:48:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:49:21] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 17.458 second response time [19:55:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:56:12] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 13.795 second response time [19:57:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:06:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 499123 MB (9% inode=43%): [20:17:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:18:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:24:21] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:40:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:41:12] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 15.094 second response time [20:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:43:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:49:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:06:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 496942 MB (9% inode=43%): [21:17:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:18:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:26:21] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 274 bytes in 20.248 second response time [21:27:22] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 17.136 second response time [21:30:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42452 MB (10% inode=99%): [21:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:44:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:49:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:52:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:53:22] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.782 second response time [21:55:22] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.213 second response time [21:58:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:06:53] / on willow is WARNING: DISK WARNING - free space: / 21972 MB (20% inode=99%): [22:06:53] /tmp on willow is WARNING: DISK WARNING - free space: / 21972 MB (20% inode=99%): [22:07:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 495268 MB (9% inode=43%): [22:17:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:18:33] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:39:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:41:22] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:41:22] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:41:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:43:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:44:12] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [22:44:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:32] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2267 [22:44:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:44:53] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2206 [22:44:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 81829 MB (8% inode=98%): [22:44:54] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 81829 MB (8% inode=98%): [22:44:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42379 MB (10% inode=99%): [22:45:02] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:45:53] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [22:47:03] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [22:49:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:50:02] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2289 [22:51:53] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [22:52:53] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2663 [22:52:53] MySQL on z-dat-s7-a is OK: Uptime: 435 Threads: 12 Questions: 1175 Slow queries: 0 Opens: 208 Flush tables: 1 Open tables: 198 Queries per second avg: 2.701 [22:53:02] MySQL slave on z-dat-s6-a is OK: Uptime: 450 Threads: 12 Questions: 44735 Slow queries: 12 Opens: 2206 Flush tables: 1 Open tables: 166 Queries per second avg: 99.411 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1724 [22:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:06:53] / on willow is WARNING: DISK WARNING - free space: / 21835 MB (20% inode=99%): [23:06:53] /tmp on willow is WARNING: DISK WARNING - free space: / 21835 MB (20% inode=99%): [23:07:21] MySQL slave on z-dat-s3-a is OK: Uptime: 303315 Threads: 17 Questions: 238668496 Slow queries: 15461 Opens: 3746217 Flush tables: 1 Open tables: 16384 Queries per second avg: 786.866 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1754 [23:08:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 486578 MB (9% inode=42%): [23:17:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:17:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:18:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:01] nacht ts [23:36:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:38:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:40:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:44:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:44:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:49:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:58:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7