[00:22:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:22:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33458 MB (5% inode=99%): [00:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:25:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:30:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 554558 MB (10% inode=46%): [00:42:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:49:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:56:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21859 MB (5% inode=99%): [00:57:43] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 115576 MB (11% inode=99%): [00:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:22:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:22:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33340 MB (5% inode=99%): [01:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:25:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:30:11] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 554453 MB (10% inode=46%): [01:42:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:58:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:22:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:22:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33230 MB (5% inode=99%): [02:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:25:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:31:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:35:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 85093 MB (8% inode=98%): [02:35:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 85093 MB (8% inode=98%): [02:35:43] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 43527 MB (10% inode=99%): [02:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552394 MB (10% inode=45%): [02:42:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:23:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:23:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33069 MB (5% inode=99%): [03:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:26:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:32:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 553231 MB (10% inode=45%): [03:42:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:50:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:23:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:23:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32938 MB (5% inode=99%): [04:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:27:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:32:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 552134 MB (10% inode=45%): [04:42:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:51:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:23:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:23:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32842 MB (5% inode=99%): [05:24:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:27:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:32:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551883 MB (10% inode=45%): [05:42:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:51:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:56:42] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 17156 MB (1% inode=96%): [05:58:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:23:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:23:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32779 MB (5% inode=99%): [06:25:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:32:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:41:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551786 MB (10% inode=45%): [06:42:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:44:43] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [06:44:44] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [06:44:44] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 43081 MB (10% inode=99%): [06:44:44] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 85317 MB (8% inode=98%): [06:44:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 85317 MB (8% inode=98%): [06:51:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:58:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:58:42] MySQL slave on z-dat-s3-a is OK: Uptime: 1255 Threads: 10 Questions: 1653 Slow queries: 1 Opens: 319 Flush tables: 1 Open tables: 304 Queries per second avg: 1.317 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1426 [06:58:42] MySQL on z-dat-s3-a is OK: Uptime: 1255 Threads: 10 Questions: 1655 Slow queries: 1 Opens: 320 Flush tables: 1 Open tables: 305 Queries per second avg: 1.318 [07:24:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:24:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32699 MB (5% inode=99%): [07:26:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:32:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551482 MB (10% inode=45%): [07:43:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:49:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:51:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:54:11] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:06:02] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:06:02] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:06:11] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:24:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32596 MB (5% inode=99%): [08:26:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:32:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:41:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551223 MB (10% inode=45%): [08:43:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:51:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:43] /sql on z-dat-s6-a is CRITICAL: Connection refused by host [09:10:43] /sql on z-dat-s7-a is CRITICAL: Connection refused by host [09:10:44] /sql on z-dat-s3-a is CRITICAL: Connection refused by host [09:19:43] SSH on hyacinth is CRITICAL: Connection refused [09:19:43] Load avg. on hyacinth is CRITICAL: Connection refused by host [09:19:43] /tmp on hyacinth is CRITICAL: Connection refused by host [09:19:43] SMTP on hyacinth is CRITICAL: Connection refused [09:19:52] RAID on hyacinth is CRITICAL: Connection refused by host [09:19:52] Environment IPMI on hyacinth is CRITICAL: Connection refused by host [09:20:03] / on hyacinth is CRITICAL: Connection refused by host [09:20:22] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 7859 MB (26% inode=85%): [09:20:42] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:20:43] Load avg. on hyacinth is OK: OK - load average: 0.94, 0.41, 0.25 [09:20:43] /tmp on hyacinth is OK: DISK OK - free space: /tmp 43886 MB (100% inode=99%): [09:20:43] SMTP on hyacinth is OK: SMTP OK - 0.268 sec. response time [09:20:51] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:21:04] / on hyacinth is OK: DISK OK - free space: / 7854 MB (26% inode=85%): [09:21:04] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [09:24:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:24:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32491 MB (5% inode=99%): [09:26:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:26:52] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [09:26:53] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [09:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:27:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:27:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:27:43] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 42163 MB (10% inode=99%): [09:27:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 85985 MB (8% inode=98%): [09:27:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 85985 MB (8% inode=98%): [09:32:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:41:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 551000 MB (10% inode=45%): [09:42:52] MySQL on z-dat-s3-a is OK: Uptime: 1344 Threads: 1 Questions: 139 Slow queries: 0 Opens: 30 Flush tables: 1 Open tables: 23 Queries per second avg: 0.103 [09:42:52] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1937 [09:44:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:51:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:52] MySQL slave on z-dat-s3-a is OK: Uptime: 2964 Threads: 15 Questions: 474482 Slow queries: 68 Opens: 11309 Flush tables: 1 Open tables: 11299 Queries per second avg: 160.81 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1632 [10:24:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:24:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32391 MB (5% inode=99%): [10:26:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:27:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:27:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:27:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:33:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:41:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550808 MB (10% inode=45%): [10:44:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:51:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:58:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:58:52] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [10:58:53] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [11:11:28] hello all [11:18:32] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2151 [11:20:52] MySQL on z-dat-s3-a is OK: Uptime: 1727 Threads: 19 Questions: 930 Slow queries: 5 Opens: 325 Flush tables: 1 Open tables: 310 Queries per second avg: 0.538 [11:20:52] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2505 [11:24:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:24:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32258 MB (5% inode=99%): [11:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:27:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:27:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:27:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:30:23] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2111 [11:33:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:40:52] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [11:41:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550630 MB (10% inode=45%): [11:44:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:46:31] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [11:52:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:06:23] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3631 [12:24:45] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:24:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32146 MB (5% inode=99%): [12:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:27:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:28:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:28:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:30:32] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3451 [12:33:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:39:32] MySQL slave on z-dat-s6-a is OK: Uptime: 11943 Threads: 27 Questions: 1198185 Slow queries: 4124 Opens: 518 Flush tables: 1 Open tables: 438 Queries per second avg: 100.325 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1701 [12:40:52] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6932 [12:41:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 549260 MB (10% inode=45%): [12:45:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:52:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:55:23] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3339 [12:57:43] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 115157 MB (11% inode=99%): [12:58:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:59:22] MySQL slave on z-dat-s7-a is OK: Uptime: 13132 Threads: 6 Questions: 5043483 Slow queries: 1538 Opens: 1461 Flush tables: 1 Open tables: 1034 Queries per second avg: 384.60 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1732 [13:24:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:25:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32042 MB (5% inode=99%): [13:26:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:28:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:28:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:28:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:34:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:40:52] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5716 [13:42:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535473 MB (10% inode=45%): [13:45:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:51:52] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3430 [13:52:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:54:48] DaBPunkt: Do you know if anyone moved some databases/database servers around? Commonsdelinker lost it's database [13:56:26] We didn't move databases. Only hycinth was rebooted in the morning. Where was commonsdelinkers db? [13:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:58:52] MySQL slave on z-dat-s3-a is OK: Uptime: 11207 Threads: 14 Questions: 3410633 Slow queries: 3999 Opens: 27013 Flush tables: 1 Open tables: 16384 Queries per second avg: 304.330 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1744 [14:00:38] multichill: I think it is a config error [14:01:02] the config is pointing to sql-s4 [14:01:08] it should probably point to sql-s4-user [14:01:12] Bryan: Ok. Who changed the config? And why are multiple tools broken? [14:01:58] I don't even know what sql-s4-user is, let alone use it [14:02:44] no, that's not it [14:03:02] ok wtf [14:04:03] DaBPunkt: it was supposed to be on sql-s4-user [14:04:14] p_delinker_p [14:08:28] DaBPunkt: did sql-s4-user originally point to hyacinth? [14:08:33] because it points now to daphne [14:08:53] and I can't connect to hyacinth [14:08:58] it should point to z-dat-s4-a. Let me look [14:09:57] ah, z-dat-s4-a looks down. Wait a moemnt [14:10:22] thanks :) [14:15:01] Bryan: Nice to see you here again. How are you doing? [14:15:17] ok, I make it fail now and marked it as down in status [14:15:52] multichill: hey [14:15:55] been busy graduating [14:16:00] but now done :) [14:16:48] nosy commented out the sql-partition of z-dat-s4-a. I have to speak with her first [14:20:13] ok [14:21:05] DaB. * [Toolserver-announce] z-dat-s4-a (s4-user) is down (was: Re: [Toolserver-l] Reboot of hyacinth s3/s6/s7) [14:21:05] DaB. * [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot of hyacinth s3/s6/s7) [14:25:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:25:44] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31932 MB (5% inode=99%): [14:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:28:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:28:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:28:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:30:07] Bryan: We're planning to do a hackathon on 10 November. Are you in? ;-) [14:30:26] depends [14:30:35] mostly on whether and where I work by then [14:34:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:42:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 536255 MB (10% inode=45%): [14:45:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:52:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:25:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:26:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:26:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31840 MB (5% inode=99%): [15:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:28:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:29:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:29:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:34:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:42:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 536640 MB (10% inode=45%): [15:45:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:52:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:58:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:25:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:34:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [16:43:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 536222 MB (10% inode=45%): [16:45:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:46:37] DaBPunkt: hi ! are you around ? I would need a bit more space for the MMP interwikibot… I believe I must fill a JIRA request ? [16:52:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:58:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:26:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:26:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31650 MB (5% inode=99%): [17:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:29:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:30:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:30:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:35:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [17:44:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535799 MB (10% inode=45%): [17:46:03] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:52:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:56:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 17297 MB (1% inode=96%): [17:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:26:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:26:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31565 MB (5% inode=99%): [18:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:29:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:30:23] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:30:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:35:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [18:45:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535541 MB (10% inode=45%): [18:46:05] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:53:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:26:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:26:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31462 MB (5% inode=99%): [19:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:30:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:31:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:31:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:36:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:37:15] the cron scheduling looks like it's in a much better shape than yesterday ^^ [19:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [19:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 535238 MB (10% inode=45%): [19:47:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:54:22] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:26:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:26:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31355 MB (5% inode=99%): [20:27:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:30:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:31:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:31:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:37:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:41:51] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [20:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 534854 MB (10% inode=45%): [20:47:04] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:54:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:26:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:26:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31257 MB (5% inode=99%): [21:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:28:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 85555 MB (8% inode=98%): [21:28:43] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 41933 MB (10% inode=99%): [21:28:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 85555 MB (8% inode=98%): [21:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:31:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:31:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [21:45:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 534590 MB (10% inode=45%): [21:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:54:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:26:32] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:26:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:26:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31150 MB (5% inode=99%): [22:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:31:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:31:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [22:45:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 534385 MB (10% inode=45%): [22:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:54:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:22] nacht ts [23:26:31] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:27:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:27:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:27:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31076 MB (5% inode=99%): [23:30:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:32:22] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:32:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:37:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:41:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [23:45:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 532968 MB (10% inode=45%): [23:47:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:54:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:58:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.