[00:05:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:10:01] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44349 MB (10% inode=99%): [00:11:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:18:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:21:40] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [00:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:23:50] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24351 MB (5% inode=99%): [00:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:36:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:42:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:42:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45745 MB (7% inode=99%): [00:49:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:04:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:06:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 27675 MB (6% inode=99%): [01:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:12:02] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 44809 MB (11% inode=99%): [01:18:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:21:40] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [01:23:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:43:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:43:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45627 MB (7% inode=99%): [01:44:02] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44716 MB (10% inode=99%): [02:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:18:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:21:42] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [02:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:35:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:44:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:44:01] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45506 MB (7% inode=99%): [03:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:18:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:21:40] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [03:23:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:35:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:44:02] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 45520 MB (11% inode=99%): [03:44:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:44:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45265 MB (7% inode=99%): [03:51:02] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44483 MB (10% inode=99%): [04:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:18:32] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:21:40] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [04:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:35:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:37:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:45:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:45:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45166 MB (7% inode=99%): [05:05:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:19:31] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [05:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:37:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:46:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45043 MB (7% inode=99%): [05:52:01] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 7510 MB (0% inode=92%): [06:05:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:19:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [06:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:46:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44878 MB (7% inode=99%): [07:05:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:11:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:19:33] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:21:40] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [07:23:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:46:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44764 MB (7% inode=99%): [08:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:19:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:20:32] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1944.000000 [08:20:51] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1939 [08:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [08:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:30:52] MySQL slave on rosemary is OK: Uptime: 13194806 Threads: 11 Questions: 5606114197 Slow queries: 1632716 Opens: 175574 Flush tables: 6 Open tables: 3805 Queries per second avg: 424.872 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [08:31:32] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1787.000000 [08:35:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:47:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:47:01] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44680 MB (7% inode=99%): [08:49:32] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1898.000000 [08:49:52] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1893 [09:00:32] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1788.000000 [09:00:51] MySQL slave on rosemary is OK: Uptime: 13196606 Threads: 10 Questions: 5607062229 Slow queries: 1633085 Opens: 175606 Flush tables: 6 Open tables: 3806 Queries per second avg: 424.886 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1758 [09:05:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:19:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [09:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:36:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:37:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23723 MB (5% inode=99%): [09:38:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25522 MB (6% inode=99%): [09:40:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106429 MB (10% inode=99%): [09:41:01] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 116210 MB (11% inode=99%): [09:47:03] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:47:04] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44607 MB (7% inode=99%): [10:05:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:15:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106877 MB (10% inode=99%): [10:16:01] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107201 MB (11% inode=99%): [10:18:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106823 MB (10% inode=99%): [10:20:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:21:40] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [10:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:36:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:36:36] @replag [10:36:37] liangent: s2-user: 1h 6m 26s [+0.03 s/s]; s2-user-c: 25m 20s [+0.01 s/s]; s3-rr-a: 2m 2s [+0.00 s/s]; s3-user: 2m 2s [+0.00 s/s]; s5-rr-a: 28s [-0.00 s/s]; s5-user: 28s [-0.00 s/s]; s5-user-c: 25m 20s [+0.01 s/s] [10:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:38:02] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 114889 MB (11% inode=99%): [10:47:41] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2163.000000 [10:48:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:48:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44506 MB (7% inode=99%): [10:57:02] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 85950 MB (8% inode=98%): [10:57:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 85950 MB (8% inode=98%): [10:59:42] s4 replag on cassia is CRITICAL: (Service Check Timed Out) [11:00:31] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2881.000000 [11:05:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:12:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [11:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [11:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:36:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:36:43] hello all [11:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:38:40] hello o/ [11:41:02] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 45613 MB (11% inode=99%): [11:48:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:49:01] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44384 MB (7% inode=99%): [12:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:12:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5812.000000 [12:19:38] re [12:20:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [12:23:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:29:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23830 MB (5% inode=99%): [12:33:30] I need more disc quota, should ask it in jira ? [12:35:14] yes [12:35:23] good :) [12:36:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:48:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:49:04] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44299 MB (7% inode=99%): [12:53:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25772 MB (6% inode=99%): [13:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:11:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:12:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5690.000000 [13:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [13:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:34:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24279 MB (5% inode=99%): [13:36:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:37:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:37:54] DaBPunkt: FYI, I got a disk full error on sql-s4-user yesterday [13:38:48] oh, only 22GB left [13:41:26] Maybe its one of those quirks tsnag complains about. Anyway, it worked immediately afterwards (well, technically 40 minutes later). [13:48:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:50:01] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44179 MB (7% inode=99%): [13:51:27] Dispenser: sounds like a temporary table. I will speak with nosy about increasing the disc-space [14:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:13:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6115.000000 [14:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [14:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:34:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 27941 MB (6% inode=99%): [14:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:48:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:48:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24187 MB (5% inode=99%): [14:50:01] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44065 MB (7% inode=99%): [15:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:11:01] Load avg. on willow is WARNING: WARNING - load average: 23.23, 17.82, 13.65 [15:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:13:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6823.000000 [15:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [15:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:40:01] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.33, 22.23, 20.03 [15:46:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 26823 MB (6% inode=99%): [15:47:02] Load avg. on willow is WARNING: WARNING - load average: 16.97, 19.88, 19.93 [15:48:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:49:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.07, 20.15, 20.02 [15:50:01] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43958 MB (7% inode=99%): [15:51:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24202 MB (5% inode=99%): [16:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:13:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7719.000000 [16:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [16:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:30:01] Load avg. on willow is WARNING: WARNING - load average: 5.57, 9.46, 19.44 [16:36:02] Load avg. on willow is OK: OK - load average: 5.65, 6.77, 14.92 [16:37:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:48:03] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:48:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29627 MB (7% inode=99%): [16:50:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43821 MB (7% inode=99%): [17:01:01] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44683 MB (10% inode=99%): [17:05:01] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 44729 MB (11% inode=99%): [17:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:12:02] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44710 MB (10% inode=99%): [17:13:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4738.000000 [17:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [17:24:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:37:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:49:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:50:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43688 MB (7% inode=99%): [17:52:02] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 6922 MB (0% inode=92%): [18:02:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24007 MB (5% inode=99%): [18:06:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:14:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4100.000000 [18:20:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [18:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:24:51] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3575.000000 [18:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:46:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25205 MB (6% inode=99%): [18:49:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:51:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43570 MB (7% inode=99%): [19:01:51] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1747.000000 [19:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:11:01] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 44905 MB (11% inode=99%): [19:11:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:18:01] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44572 MB (10% inode=99%): [19:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [19:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:37:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:37:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:49:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:51:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43442 MB (7% inode=99%): [19:57:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106830 MB (10% inode=99%): [20:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:11:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:12:02] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 117041 MB (12% inode=99%): [20:20:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [20:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:37:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:49:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:52:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43343 MB (7% inode=99%): [21:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:11:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:20:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [21:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:37:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:37:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:49:02] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:52:01] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43227 MB (7% inode=99%): [22:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:20:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:21:40] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [22:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:37:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:38:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:46:51] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24146 MB (5% inode=99%): [22:50:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:52:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43079 MB (7% inode=99%): [22:58:01] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 88027 MB (9% inode=98%): [22:58:01] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 88027 MB (9% inode=98%): [23:06:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:11:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:20:41] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:21:41] APT on yarrow is WARNING: APT WARNING: 4 packages available for upgrade (0 critical updates). [23:24:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:27:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 27445 MB (6% inode=99%): [23:37:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:38:35] nacht ts [23:38:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:51:01] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:52:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43006 MB (7% inode=99%):