[00:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:02:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:03:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:07:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3631 [00:08:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [00:12:08] any admin around? [00:15:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:43:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31313 MB (5% inode=99%): [00:49:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:59:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:00:58] Free Memory on turnera is WARNING: WARNING - 5.5% (463344 kB) free! [01:02:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:03:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:05:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (325660 kB) free! [01:07:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5489 [01:08:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5490.000000 [01:15:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:32:38] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:33:09] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 74784 MB (7% inode=98%): [01:35:58] Free Memory on turnera is WARNING: WARNING - 5.6% (473344 kB) free! [01:42:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (405468 kB) free! [01:43:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:45:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31246 MB (5% inode=99%): [01:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:49:58] Free Memory on turnera is WARNING: WARNING - 5.4% (452020 kB) free! [01:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:59:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:02:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.0% (335872 kB) free! [02:02:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:03:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:08:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6730 [02:09:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6745.000000 [02:15:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:20:58] Free Memory on turnera is WARNING: WARNING - 5.1% (429332 kB) free! [02:33:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.2% (350852 kB) free! [02:43:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:45:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:46:58] Free Memory on turnera is WARNING: WARNING - 5.1% (424124 kB) free! [02:47:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (418364 kB) free! [02:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31189 MB (5% inode=99%): [02:49:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:59:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:02:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:04:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:08:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 465198 MB (8% inode=41%): [03:09:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7711.000000 [03:09:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7712 [03:16:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:29:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:43:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:43:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (371868 kB) free! [03:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:47:59] Free Memory on turnera is WARNING: WARNING - 5.1% (429140 kB) free! [03:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31131 MB (5% inode=99%): [03:49:39] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:59:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:59:28] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:00:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:00:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:01:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (248748 kB) free! [04:03:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:04:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:05:58] Free Memory on turnera is OK: OK - 9.7% (813888 kB) free. [04:08:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 465066 MB (8% inode=41%): [04:09:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7875.000000 [04:09:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7840 [04:16:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:39:58] Free Memory on turnera is WARNING: WARNING - 6.6% (556152 kB) free! [04:44:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:45:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417768 kB) free! [04:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:46:59] Free Memory on turnera is WARNING: WARNING - 5.1% (424244 kB) free! [04:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31108 MB (5% inode=99%): [04:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:59:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:01:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:01:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:03:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:04:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:08:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 464947 MB (8% inode=41%): [05:09:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9693 [05:10:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9718.000000 [05:16:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:17:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.8% (317444 kB) free! [05:42:58] Free Memory on turnera is OK: OK - 7.2% (602988 kB) free. [05:44:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 31061 MB (5% inode=99%): [05:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:51:58] Free Memory on turnera is WARNING: WARNING - 5.9% (493512 kB) free! [05:56:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (386116 kB) free! [05:59:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:01:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:01:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:03:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:04:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:05:58] Free Memory on turnera is WARNING: WARNING - 6.6% (552808 kB) free! [06:08:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 464860 MB (8% inode=41%): [06:10:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10889.000000 [06:10:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10903 [06:11:59] Free Memory on turnera is OK: OK - 7.2% (603484 kB) free. [06:16:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:44:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30994 MB (5% inode=99%): [06:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:50:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (391428 kB) free! [06:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:59:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:02:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:02:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:04:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:05:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:09:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 464766 MB (8% inode=41%): [07:10:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13281.000000 [07:10:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13300 [07:16:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:26:59] Free Memory on turnera is WARNING: WARNING - 5.1% (425960 kB) free! [07:27:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422884 kB) free! [07:28:58] Free Memory on turnera is WARNING: WARNING - 5.1% (428028 kB) free! [07:45:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:48:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (376280 kB) free! [07:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30790 MB (5% inode=99%): [07:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:57:25] Marlen Caemmerer * Re: [Toolserver-l] /mnt/user-store is away [07:59:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:00:22] thank you nosy [08:02:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:02:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:04:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:05:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:09:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 464603 MB (8% inode=41%): [08:10:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13535.000000 [08:10:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13547 [08:16:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:46:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30559 MB (4% inode=99%): [08:49:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (405356 kB) free! [08:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:50:58] Free Memory on turnera is WARNING: WARNING - 5.1% (424332 kB) free! [08:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:51:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (423056 kB) free! [08:52:58] Free Memory on turnera is WARNING: WARNING - 5.2% (436392 kB) free! [09:00:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:02:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:02:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:04:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (408344 kB) free! [09:04:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:05:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13413.000000 [09:10:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 464349 MB (8% inode=41%): [09:10:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13403 [09:11:48] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39388 MB (9% inode=99%): [09:12:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 72729 MB (7% inode=98%): [09:14:58] Free Memory on turnera is WARNING: WARNING - 5.1% (431524 kB) free! [09:16:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:22:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (415108 kB) free! [09:29:58] Free Memory on turnera is WARNING: WARNING - 5.1% (424516 kB) free! [09:45:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2055.000000 [09:46:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:48:59] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2132.000000 [09:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30367 MB (4% inode=99%): [09:49:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (301404 kB) free! [09:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:00:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:02:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:02:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:04:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:05:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:10:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12671.000000 [10:10:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463094 MB (8% inode=41%): [10:10:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12642 [10:16:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:20:58] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [10:20:58] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:20:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [10:27:09] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:09] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:27:19] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:27:19] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:27:19] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:27:28] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:27:47] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:36:38] Load avg. on hemlock is CRITICAL: Connection refused by host [10:36:58] /aux0 on hemlock is CRITICAL: Connection refused by host [10:37:08] CAM on hemlock is CRITICAL: Connection refused by host [10:37:17] /tmp on hemlock is CRITICAL: Connection refused by host [10:37:18] /home on hemlock is CRITICAL: Connection refused by host [10:37:18] / on hemlock is CRITICAL: Connection refused by host [10:37:19] Environment IPMI on hemlock is CRITICAL: Connection refused by host [10:37:58] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:38:09] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:38:18] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:38:18] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:38:28] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:38:28] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:38:47] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:40:58] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:44:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 87315 MB (8% inode=99%): [10:46:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 30084 MB (4% inode=99%): [10:49:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.3% (279572 kB) free! [10:50:08] CAM on hemlock is OK: OK - cam detected no new errors [10:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:53:19] /tmp on hemlock is OK: DISK OK - free space: / 6042 MB (30% inode=89%): [10:54:07] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:54:38] Load avg. on hemlock is OK: OK - load average: 0.41, 0.29, 0.27 [10:57:59] /aux0 on hemlock is CRITICAL: Connection refused by host [10:58:18] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1645 MB (10% inode=99%): [10:58:18] Environment IPMI on hemlock is CRITICAL: Connection refused by host [10:58:58] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:59:28] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:00:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:02:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:03:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:04:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:05:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:28] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [11:09:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462039 MB (8% inode=41%): [11:10:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12689.000000 [11:11:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12669 [11:16:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:20:58] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5043.000000 [11:21:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5432.000000 [11:29:59] /aux0 on hemlock is CRITICAL: Connection refused by host [11:35:19] /tmp on hemlock is CRITICAL: Connection refused by host [11:35:19] /home on hemlock is CRITICAL: Connection refused by host [11:35:19] / on hemlock is CRITICAL: Connection refused by host [11:35:19] Environment IPMI on hemlock is CRITICAL: Connection refused by host [11:35:38] Load avg. on hemlock is CRITICAL: Connection refused by host [11:36:09] CAM on hemlock is CRITICAL: Connection refused by host [11:40:19] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1787 MB (11% inode=99%): [11:41:18] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1584 MB (9% inode=99%): [11:43:17] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1806 MB (11% inode=99%): [11:46:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:48:08] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29809 MB (4% inode=99%): [11:49:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.6% (303052 kB) free! [11:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:51:26] DaB. * [Toolserver-announce] Fwd: Re: [Toolserver-l] /mnt/user-store is away [11:58:58] Free Memory on turnera is WARNING: WARNING - 5.2% (438876 kB) free! [11:59:58] Free Memory on turnera is OK: OK - 10.4% (875252 kB) free. [11:59:58] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:00:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:02:08] CAM on hemlock is OK: OK - cam detected no new errors [12:02:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:02:17] /home on hemlock is OK: DISK OK - free space: /home 18977 MB (37% inode=87%): [12:02:18] /tmp on hemlock is OK: DISK OK - free space: / 6040 MB (30% inode=89%): [12:02:18] / on hemlock is OK: DISK OK - free space: / 6040 MB (30% inode=89%): [12:02:28] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [12:02:38] Load avg. on hemlock is OK: OK - load average: 1.42, 0.98, 0.52 [12:02:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461862 MB (8% inode=41%): [12:03:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:05:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:05:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:08:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.6% (216948 kB) free! [12:10:37] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13433.000000 [12:12:58] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13399 [12:13:19] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1047 MB (6% inode=99%): [12:13:58] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2172 [12:16:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:18:19] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2081 MB (12% inode=99%): [12:19:18] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1249 MB (7% inode=99%): [12:20:58] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4464.000000 [12:21:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7923.000000 [12:26:58] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [12:32:47] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:33:08] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:33:19] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:33:58] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [12:37:28] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:37:38] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:18] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:48] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39158 MB (9% inode=99%): [12:38:59] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:59] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:59] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73153 MB (7% inode=98%): [12:39:09] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:09] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:09] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:09] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:39:18] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1883 MB (11% inode=99%): [12:39:28] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73153 MB (7% inode=98%): [12:39:28] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:39:47] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:39:58] Free Memory on turnera is WARNING: WARNING - 5.2% (434964 kB) free! [12:39:59] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:39:59] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:39:59] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:39:59] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:40:28] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2041 [12:40:48] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [12:42:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (410788 kB) free! [12:44:58] Free Memory on turnera is WARNING: WARNING - 5.1% (423324 kB) free! [12:46:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:46:37] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [12:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:46:47] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [12:47:48] MySQL on z-dat-s7-a is OK: Uptime: 460 Threads: 2 Questions: 8 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 13 Queries per second avg: 0.17 [12:47:48] MySQL slave on z-dat-s7-a is OK: Uptime: 460 Threads: 2 Questions: 8 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 13 Queries per second avg: 0.17 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29536 MB (4% inode=99%): [12:49:58] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1784.000000 [12:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:51:28] MySQL slave on z-dat-s6-a is OK: Uptime: 272474 Threads: 6 Questions: 52092046 Slow queries: 20921 Opens: 556442 Flush tables: 1 Open tables: 2792 Queries per second avg: 191.181 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1595 [12:52:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [12:52:48] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:08] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [12:53:18] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:02:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:02:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461729 MB (8% inode=41%): [13:03:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:05:37] MySQL on z-dat-s3-a is OK: Uptime: 1528 Threads: 4 Questions: 1424 Slow queries: 0 Opens: 238 Flush tables: 1 Open tables: 229 Queries per second avg: 0.931 [13:05:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:05:59] Free Memory on turnera is WARNING: WARNING - 5.8% (485988 kB) free! [13:05:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:09:53] nosy, can you take a look at the web servers? [13:10:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9628.000000 [13:12:59] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9206 [13:16:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:21:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10821.000000 [13:26:48] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [13:30:52] * Nettrom seconds Platonides requeset [13:30:56] request, even [13:30:58] Free Memory on turnera is OK: OK - 7.1% (593776 kB) free. [13:34:47] MySQL slave on z-dat-s3-a is OK: Uptime: 3279 Threads: 9 Questions: 858788 Slow queries: 3 Opens: 15058 Flush tables: 1 Open tables: 7498 Queries per second avg: 261.905 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1798 [13:39:19] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2512 MB (14% inode=99%): [13:46:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 29224 MB (4% inode=99%): [13:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:51:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:52:48] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:09] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [13:53:18] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:18] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [14:00:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:02:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:02:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461729 MB (8% inode=41%): [14:03:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:05:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:05:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:10:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3733.000000 [14:11:38] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3597.000000 [14:11:58] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3525 [14:16:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:21:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12406.000000 [14:27:15] I keep getting 504 errors when I try to run tools. Occassionally 500 or 502, as well. I can't even pull up the channel log to see if you discussed this already. [14:29:11] Platonides pinged nosy about it at 15:09 UTC [14:29:18] JIRA seems to be up, doesn't seem to be a ticket about it there though [14:30:53] sorry, the UTC time might be wrong [14:31:07] that's likely CET, given my connection [14:31:29] 90 mins ago, either way [14:32:44] cest, even [14:34:04] Are there still issues with toolservers filesystem? [14:34:31] apparently [14:35:58] MySQL slave on rosemary is OK: Uptime: 15031115 Threads: 10 Questions: 6679955013 Slow queries: 2105877 Opens: 291213 Flush tables: 6 Open tables: 3876 Queries per second avg: 444.408 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1773 [14:36:38] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1693.000000 [14:39:19] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2331 MB (13% inode=99%): [14:46:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:49:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28984 MB (4% inode=99%): [14:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:52:47] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:08] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [14:53:19] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/2:DOWN: 1 int NOK : CRITICAL [15:00:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:02:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:02:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461729 MB (8% inode=41%): [15:03:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:06:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:06:59] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:10:29] /aux0 on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:17:58] Free Memory on turnera is WARNING: WARNING - 5.5% (462036 kB) free! [15:22:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8644.000000 [15:38:39] hello all [15:38:51] It looks like we have another hemlock disc-problem [15:39:18] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2358 MB (14% inode=99%): [15:39:45] yes [15:39:51] webservers down [15:40:05] 258 instances of "df -k /mnt/user-store" in ortelius [15:40:16] ran by root, that doesn't look right [15:40:39] 260 now, same in wolfsbane [15:40:53] maybe created by nagios :/ [15:40:58] I guess that's nagios-discs checks [15:41:19] it should kill them [15:41:34] but it can be looked later [15:41:58] Free Memory on turnera is OK: OK - 7.1% (596692 kB) free. [15:42:09] I will reboot hemlock [15:46:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:49:09] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is OK: FC port 0/4:UP:1 UP: OK [15:49:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28821 MB (4% inode=99%): [15:49:28] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/2:UP:1 UP: OK [15:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:51:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:52:48] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3469.000000 [15:53:18] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:38] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.021 second response time [15:54:09] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.018 second response time [15:57:09] finally [15:58:49] Platonides: there is a holiday today in germany [15:59:11] I was refering to the server, not to you [15:59:23] Platonides: I got that :) [15:59:44] although it's true I didn't know it was a holiday today :) [16:00:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:01:12] was hemlock showing some problem this time? [16:01:24] I guess you still don't know why the web servers don't like it [16:03:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:04:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:04:09] hemlock is complaining from time to time that a "number of links reached vx_maxlink"-problem occures, but I'm unsure if it is related. [16:04:26] I have to leave for making dinner now. cu [16:05:22] DaBPunkt: thanks for fixing this, appreciate it, have a good meal! :) [16:06:57] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (412332 kB) free! [16:07:10] DaBPunkt: the files in user-store are not showing atm [16:07:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:07:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:08:57] Free Memory on turnera is WARNING: WARNING - 5.2% (437020 kB) free! [16:09:29] is it a VxFS filesystem ? [16:09:46] I think that error would be that a ln command failed [16:09:56] which shouldn't really be a problem [16:10:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (419312 kB) free! [16:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:23:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [16:29:59] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:33:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3517.000000 [16:38:42] giftpflanze: yes [16:40:19] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2298 MB (13% inode=99%): [16:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:47:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:50:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28580 MB (4% inode=99%): [16:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:52:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:53:57] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (357468 kB) free! [16:54:29] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:58:28] usr store gone again :-/ [16:58:50] is that hw or sw issue? [17:01:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:01:58] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1749.000000 [17:03:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:05:04] Hi. Ist es möglich, das Jira vom Toolserver per Mylyn in Eclipse einzubinden? [17:05:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:05:49] wel... or in English: Is it possible/how do I connect the Toolserver Jira to Eclipse (via Mylyn)? [17:07:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:07:58] Free Memory on turnera is WARNING: WARNING - 5.1% (427036 kB) free! [17:07:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:11:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (384568 kB) free! [17:14:57] Free Memory on turnera is WARNING: WARNING - 5.1% (429776 kB) free! [17:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:18:18] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1486 MB (9% inode=99%): [17:19:19] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3053 MB (17% inode=99%): [17:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:51:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28379 MB (4% inode=99%): [17:52:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:01:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:03:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:05:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:07:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:07:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:12:54] anybody here with knowledge about the maps stuff (especially osm_mapnik database) on the toolserver? [18:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:19:18] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2284 MB (13% inode=99%): [18:40:33] any idea why I get a bad gateway error repeatingly with the lalm (java application)? [18:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:51:09] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 28137 MB (4% inode=99%): [18:52:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:53:00] trying to find all pages in article namespace created by a user, here is my query: [18:53:02] SELECT COUNT(DISTINCT page_title) FROM page JOIN revision ON page_id=rev_page WHERE rev_user_text="Scottywong" AND page_namespace=0 AND page_is_redirect=0 AND rev_parent_id=0; [18:53:44] it takes 3-5 minutes to complete, especially for users with high edit counts. anyone know why it takes so long, and how to make it more efficient? [18:54:19] Snottywong: please notice that this qurey will show too few results [18:54:32] rev_parent_id was not back-populatted AFAIK [18:54:38] oh [18:55:14] is there a better way to find new articles created by a user? I'm not that concerned if it's not accurate all the way back to 2001. [18:55:24] DaBPunkt: when does the toolserver return a 502 (bad gateway) for a java application? Am I right that it's out of the applications scope? [18:55:59] any eta when usr-store will be back again? [18:56:27] Snottywong: there is no fast way to do that [18:57:12] Danny_B|backup: I have to speak with nosy first. I would like to copy its content to the SAN. [18:57:14] any way under 30 seconds? doesn't need to be super fast, but 5 minutes isn't going to work [18:57:21] Snottywong: no way [18:58:07] for big wikis a accurated result is in the range >30minutes [18:58:10] let me dig in my stuff for the different query i used [18:58:40] iianm it was using something like min(rev_id) [18:59:59] you can ise min(rev_id) (or min(rev_timestamp) if you like to get the first edition of a arictle – that's quite fast [19:00:31] but that's not working for "all first editions of a user" [19:00:36] hmm ok that's an interesting idea [19:00:38] of course I could be wrong :) [19:00:55] X's / TParis' tool seems to do it relatively quickly [19:01:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:02:24] Snottywong: do you need query returning just the number or can it be query returning all those articles so you can use eg. mysql_num_rows? [19:02:42] all I need is the number [19:03:02] SELECT page_title, MIN(rev_timestamp), rev_user_text FROM revision JOIN page ON rev_page = page_id WHERE page_is_redirect = 0 AND page_namespace = 0 GROUP BY rev_page HAVING rev_user_text = ""; [19:03:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:03:09] this returns all articles [19:03:20] thanks! I'll try it out [19:03:36] idk if it will wor won't be faster [19:03:45] my poor server… [19:03:46] i never ran anything with rev_parent_id [19:03:55] DaBPunkt: which one? ;-) [19:04:06] HAVING and many results… [19:04:42] Danny_B|backup: wouldn't that return the first edit that the user made to every unique article he's every edited? [19:04:49] we need more sql servers [19:05:05] so we can time to time run even long queries [19:05:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:05:17] such as the real count of articles on english wikipedia [19:05:27] I guess then you'd have to compare the rev_timestamp of those revisions to the min(rev_timestamp) for the page? [19:05:35] (when it was about to turn to 2M, the query took 25 hrs) [19:06:27] ←away from irc [19:06:46] Snottywong: as i understand that (i got that query from someone else) it takes all first revisions of all pages and then filters them by username [19:07:00] so the result will be created pages by given user [19:07:45] hmm ok, you might be right, I'll give it a go [19:07:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:07:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:08:19] /tmp on ortelius is OK: DISK OK - free space: /tmp 8065 MB (48% inode=99%): [19:09:59] Free Memory on turnera is WARNING: WARNING - 6.4% (536728 kB) free! [19:15:58] Free Memory on turnera is OK: OK - 7.1% (592816 kB) free. [19:17:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:24:18] / on damiana is WARNING: DISK WARNING - free space: / 14956 MB (20% inode=95%): [19:28:57] jongleur: Ptolemy is missing the /tiles directory (which probably shouldn't effect you) and something might be up with the database earlier on. Not sure if that is fixed by now [19:29:31] thanks, I'll see [19:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:48:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:48:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1964.000000 [19:49:07] DaBPunkt: you actually may like to send the info to the list about userstore, since the last info there was, that it is back up [19:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:51:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27961 MB (4% inode=99%): [19:53:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:55:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90511 MB (14% inode=99%): [20:01:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:03:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:05:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:07:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:07:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:08:58] Free Memory on turnera is WARNING: WARNING - 5.9% (490636 kB) free! [20:13:58] Free Memory on turnera is OK: OK - 7.4% (622648 kB) free. [20:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:24:19] / on damiana is WARNING: DISK WARNING - free space: / 12990 MB (18% inode=95%): [20:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:48:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3386.000000 [20:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:54:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:55:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90238 MB (14% inode=99%): [21:01:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:03:09] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:04:58] Free Memory on turnera is WARNING: WARNING - 6.7% (565252 kB) free! [21:05:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:07:58] Free Memory on turnera is OK: OK - 7.1% (591760 kB) free. [21:08:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:08:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:14:59] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (414824 kB) free! [21:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:25:18] / on damiana is WARNING: DISK WARNING - free space: / 10674 MB (14% inode=95%): [21:34:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3647.000000 [21:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:50:37] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:53:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [21:54:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:56:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 90019 MB (14% inode=99%): [22:02:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:03:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:05:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:08:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:08:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:17:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:23:58] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1767.000000 [22:25:18] / on damiana is WARNING: DISK WARNING - free space: / 8817 MB (12% inode=95%): [22:44:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 105188 MB (10% inode=99%): [22:46:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:48:18] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:54:19] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:57:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89820 MB (14% inode=99%): [23:02:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:03:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:04:58] Free Memory on turnera is WARNING: WARNING - 6.4% (536512 kB) free! [23:06:09] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:08:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:08:58] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:14:59] Free Memory on turnera is OK: OK - 7.1% (596876 kB) free. [23:17:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:25:19] / on damiana is CRITICAL: DISK CRITICAL - free space: / 7897 MB (10% inode=95%): [23:46:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:48:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:50:38] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:54:18] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:56:18] FC 0/4 [hemlock] on fsw2-n1-oe16-esams.mgmt is CRITICAL: FC port 0/4:DOWN: 1 int NOK : CRITICAL [23:57:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 89672 MB (14% inode=99%):