[00:13:56] nosy: "Table 'nowiki_p.categorylinks' doesn't exist" :-| [00:15:53] Not that _I_ care about [00:20:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52427.000000 [00:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27149.000000 [00:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27175 [00:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:44:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:52:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52391 MB (8% inode=99%): [00:55:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 39.59, 39.23, 39.75 [00:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:03:51] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:20:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51900.000000 [01:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29137.000000 [01:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29160 [01:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:52:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50747 MB (8% inode=99%): [01:55:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 31.09, 32.18, 34.54 [01:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:20:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48222.000000 [02:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30457.000000 [02:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30448 [02:31:32] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92881 MB (9% inode=98%): [02:31:32] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92881 MB (9% inode=98%): [02:33:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51647 MB (8% inode=99%): [02:55:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.85, 34.34, 35.21 [02:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:03:43] @replag [03:03:44] Dispenser: s1-rr-a: 8h 43m 40s [+0.49 s/s]; s1-user: 8h 43m 40s [+0.49 s/s]; s2-user: 9h 17m 32s [+0.24 s/s]; s2-user-c: 36s [-0.44 s/s]; s3-rr-a: 4m 53s [+0.01 s/s]; s3-user: 4m 53s [+0.01 s/s]; s5-user-c: 36s [-0.44 s/s] [03:20:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47654.000000 [03:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32622.000000 [03:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32645 [03:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:51:34] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 85 MB (15% inode=98%): [03:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51494 MB (8% inode=99%): [03:55:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:56:32] /tmp on hemlock is OK: DISK OK - free space: /tmp 177 MB (27% inode=98%): [03:57:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.10, 29.77, 30.06 [03:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:05:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:07:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:08:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:09:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:09:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:09:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:09:32] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:09:41] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:14:32] Free Memory on damiana is WARNING: WARNING - 5.6% (469540 kB) free! [04:18:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (414652 kB) free! [04:19:32] Free Memory on damiana is WARNING: WARNING - 5.1% (424072 kB) free! [04:20:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49506.000000 [04:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34560.000000 [04:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34562 [04:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:34:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:53:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51261 MB (8% inode=99%): [04:56:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:58:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 18.25, 18.94, 22.26 [04:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:05:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:07:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:08:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:09:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:09:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:09:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:09:32] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:09:41] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:16:41] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:16:51] CAM on hemlock is OK: OK - cam detected no new errors [05:16:51] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [05:17:22] / on hemlock is OK: DISK OK - free space: / 7033 MB (35% inode=89%): [05:17:32] Load avg. on hemlock is OK: OK - load average: 0.25, 0.23, 0.20 [05:17:32] /aux0 on hemlock is OK: DISK OK - free space: /aux0 734839 MB (13% inode=53%): [05:17:32] /home on hemlock is OK: DISK OK - free space: /home 21303 MB (42% inode=88%): [05:17:33] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 66 MB (12% inode=98%): [05:20:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46037.000000 [05:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36202.000000 [05:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36222 [05:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:34:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:39:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:41:32] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 101 MB (17% inode=98%): [05:45:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:48:32] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 17356 MB (1% inode=96%): [05:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51086 MB (8% inode=99%): [05:54:32] Load avg. on ortelius is WARNING: WARNING - load average: 15.86, 17.85, 19.80 [05:55:31] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 20.71, 18.86, 20.03 [05:56:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:02:32] Load avg. on ortelius is WARNING: WARNING - load average: 15.48, 19.06, 19.97 [06:20:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41666.000000 [06:22:33] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.86, 26.63, 23.68 [06:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36614.000000 [06:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36615 [06:33:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:35:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:53:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50980 MB (8% inode=99%): [06:56:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:05:07] Kai Krueger * [Toolserver-l] Compiling an apache module on solaris, with or without stack-protection [07:20:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41908.000000 [07:22:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 19.84, 21.04, 21.70 [07:22:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:26:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39774.000000 [07:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39794 [07:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:35:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:36:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24165 MB (5% inode=99%): [07:52:33] Free Memory on damiana is WARNING: WARNING - 5.4% (454584 kB) free! [07:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50846 MB (8% inode=99%): [07:56:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:59:31] Free Memory on damiana is OK: OK - 7.2% (601052 kB) free. [08:18:32] Load avg. on ortelius is WARNING: WARNING - load average: 16.18, 18.22, 19.99 [08:19:42] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1854.000000 [08:20:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39174.000000 [08:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40398.000000 [08:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40417 [08:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:35:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:37:42] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1701.000000 [08:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50717 MB (8% inode=99%): [08:54:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 31209 MB (7% inode=99%): [08:56:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:01:32] Load avg. on ortelius is OK: OK - load average: 9.14, 12.85, 14.84 [09:10:33] Load avg. on ortelius is WARNING: WARNING - load average: 16.86, 16.78, 15.86 [09:20:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38623.000000 [09:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:22:42] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2092.000000 [09:26:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42167.000000 [09:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42171 [09:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:34:32] Load avg. on ortelius is OK: OK - load average: 7.47, 11.78, 14.84 [09:35:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:46:43] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:33] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:33] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:33] SSH on hemlock is CRITICAL: Server answer: [09:52:34] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:52:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50505 MB (8% inode=99%): [09:54:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (339836 kB) free! [09:56:33] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:57:32] Free Memory on damiana is WARNING: WARNING - 5.6% (466972 kB) free! [09:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [10:09:32] Free Memory on damiana is OK: OK - 8.2% (689596 kB) free. [10:20:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37532.000000 [10:21:32] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:22:02] /aux0 on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:22:21] / on hemlock is OK: DISK OK - free space: / 7022 MB (35% inode=89%): [10:22:32] Load avg. on hemlock is OK: OK - load average: 0.15, 0.16, 0.29 [10:22:32] /aux0 on hemlock is OK: DISK OK - free space: /aux0 736996 MB (13% inode=53%): [10:22:32] /home on hemlock is OK: DISK OK - free space: /home 21315 MB (42% inode=88%): [10:22:41] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 64 MB (12% inode=98%): [10:22:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:22:52] CAM on hemlock is OK: OK - cam detected no new errors [10:22:52] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [10:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42074 [10:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42072.000000 [10:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:33:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:35:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:36:42] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:37:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:39:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:39:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:39:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:39:33] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:39:41] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:39:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:48:32] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:33] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (351420 kB) free! [10:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50283 MB (8% inode=99%): [10:54:01] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:54:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91211 MB (9% inode=98%): [10:55:32] Free Memory on damiana is WARNING: WARNING - 5.3% (444388 kB) free! [10:56:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:56:33] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (412244 kB) free! [10:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5862.000000 [11:00:42] Sun Grid Engine execd on wolfsbane is CRITICAL: NRPE: Unable to read output [11:01:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:06:42] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 71 MB (9% inode=99%): [11:07:42] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:32] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.079 second response time [11:09:41] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:10:33] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 611 MB (46% inode=99%): [11:21:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33506.000000 [11:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41714 [11:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41713.000000 [11:26:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 122449 MB (12% inode=99%): [11:29:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:33:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:35:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:39:33] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:39:33] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:39:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:39:33] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:39:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:39:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [11:40:32] SSH on hemlock is CRITICAL: Server answer: [11:48:33] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:31] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 50047 MB (8% inode=99%): [11:56:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9229.000000 [12:18:44] db is quite slow? [12:18:46] | 144851204 | liangent | damiana-bge0.esi.toolserver.org:50431 | zhwiki_p | Query | 159 | Sending data | SELECT /* SLOW_OK */ /* wfAutopromoteConditions 127.0.0.1 */ rev_id FROM `revision` WHERE (rev_timestamp > 20120604121545) AND rev_user = '504154' LIMIT 90 | [12:19:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24236 MB (5% inode=99%): [12:21:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28224.000000 [12:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43344 [12:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43345.000000 [12:29:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:33:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:35:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:39:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:39:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:39:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:40:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:40:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:40:32] SSH on hemlock is CRITICAL: Server answer: [12:40:32] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:48:32] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49743 MB (8% inode=99%): [12:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11097.000000 [13:08:07] Platonides * Re: [Toolserver-l] Compiling an apache module on solaris, with or without stack-protection [13:21:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25491.000000 [13:22:21] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41388 MB (10% inode=99%): [13:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43221 [13:26:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43218.000000 [13:29:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:32:42] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:33:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:33:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:34:41] Sun Grid Engine execd on wolfsbane is UNKNOWN: NRPE: Call to fork() failed [13:35:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:36:27] wolfsbane seems very bad [13:36:32] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 77 MB (9% inode=99%): [13:36:34] -bash: fork: Not enough space [13:36:50] ll /tmp/ [13:36:50] *** libc thread failure: cannot allocate thread hash table [13:36:50] Abort (core dumped) [13:36:50] -bash: fork: Not enough space [13:37:08] nosy, are you here? [13:37:41] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:38:41] /tmp on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:38:56] wolfsbane's /tmp is only 770 MB [13:39:17] quentinv57's SUL tool doesn't work [13:39:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:39:41] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 76 MB (9% inode=99%): [13:39:50] Do you have the same problem? [13:39:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:39:51] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:39:53] MrAjedrez, it's probably wolfsbane [13:40:05] try using http://ortelius.toolserver.org as hostname [13:40:17] until this gets fixed [13:40:31] Ok, thanks [13:40:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:40:32] SSH on hemlock is CRITICAL: Server answer: [13:40:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:40:33] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:40:51] Dschwen's using 650 MB of /tmp [13:42:14] those vips folders are big [13:42:28] er.. no, just files [13:43:13] still, it should be able to run with 75MB in /tmp [13:45:40] why is mapniktile3 running in wolfsbane? [13:46:20] Sent dschwen an email [13:48:32] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:41] toolserver.org HTTP on wolfsbane is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.725 second response time [13:49:42] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.014 second response time [13:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49470 MB (8% inode=99%): [13:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:57:51] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [13:58:32] Load avg. on wolfsbane is CRITICAL: Connection refused by host [13:58:32] / on wolfsbane is CRITICAL: Connection refused by host [13:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13011.000000 [14:02:32] Load avg. on wolfsbane is OK: OK - load average: 4.36, 2.74, 1.82 [14:02:32] / on wolfsbane is OK: DISK OK - free space: / 16734 MB (55% inode=93%): [14:03:01] Environment IPMI on wolfsbane is OK: ok: temperature ok fan ok voltage ok chassis ok [14:12:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:16:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24006 MB (5% inode=99%): [14:21:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23948.000000 [14:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43186 [14:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43182.000000 [14:30:42] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:31:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:32:32] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93403 MB (9% inode=98%): [14:33:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:33:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:36:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:39:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:39:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:40:31] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:40:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:40:32] SSH on hemlock is CRITICAL: Server answer: [14:40:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:40:32] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:48:32] Load avg. on daphne is WARNING: WARNING - load average: 15.26, 15.05, 13.89 [14:48:32] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49167 MB (8% inode=99%): [14:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:59:32] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 65 MB (8% inode=99%): [14:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14985.000000 [15:00:42] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:01:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:02:42] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:02:52] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:32] Load avg. on daphne is OK: OK - load average: 14.59, 14.96, 14.63 [15:07:31] I'm getting an Internal server error on the toolserver. [15:07:42] /tmp on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:07:53] that's wolfsbane problems [15:08:41] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 66 MB (8% inode=99%): [15:21:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23914.000000 [15:25:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 31605 MB (7% inode=99%): [15:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43792 [15:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43795.000000 [15:29:31] Load avg. on daphne is WARNING: WARNING - load average: 15.04, 15.08, 14.89 [15:30:32] Load avg. on daphne is OK: OK - load average: 14.77, 15.00, 14.87 [15:32:32] Load avg. on wolfsbane is CRITICAL: Connection refused by host [15:32:32] / on wolfsbane is CRITICAL: Connection refused by host [15:32:51] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [15:33:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:35:07] emijrp * Re: [Toolserver-l] Web services are down [15:36:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:38:07] Magnus Manske * Re: [Toolserver-l] Web services are down [15:39:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:39:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:40:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:40:33] SSH on hemlock is CRITICAL: Server answer: [15:40:33] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:40:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:40:33] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:43:07] Platonides * Re: [Toolserver-l] Web services are down [15:46:33] /tmp on wolfsbane is CRITICAL: Connection refused by host [15:46:42] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [15:48:32] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48996 MB (8% inode=99%): [15:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16272.000000 [16:02:51] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:41] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.068 second response time [16:12:52] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:41] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.091 second response time [16:21:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24518.000000 [16:26:07] Ilmari Karonen * [Toolserver-l] Why does Perl use so much more vmem on Linux than on Solaris? [16:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44742 [16:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44736.000000 [16:32:32] Load avg. on wolfsbane is CRITICAL: Connection refused by host [16:32:32] / on wolfsbane is CRITICAL: Connection refused by host [16:32:52] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [16:33:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:36:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:39:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:39:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:40:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:40:32] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:40:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:40:42] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:40:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:43:22] @replag [16:43:24] liangent: s1-rr-a: 12h 23m 26s [+0.10 s/s]; s1-user: 12h 23m 26s [+0.10 s/s]; s2-user: 14h 12m 25s [+0.55 s/s]; s2-user-c: 4h 49m 58s [+0.57 s/s]; s3-rr-a: 2m 11s [+0.00 s/s]; s3-user: 2m 11s [+0.00 s/s]; s5-user-c: 4h 49m 58s [+0.57 s/s]; s6-rr-a: 25s [+0.00 s/s] [16:43:25] liangent: s6-user: 25s [+0.00 s/s] [16:46:32] /tmp on wolfsbane is CRITICAL: Connection refused by host [16:46:41] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [16:48:32] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48701 MB (7% inode=99%): [16:57:31] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17883.000000 [17:01:07] ralf * Re: [Toolserver-l] Why does Perl use so much more vmem on Linux than on Solaris? [17:03:32] Load avg. on wolfsbane is OK: OK - load average: 1.43, 1.51, 1.25 [17:03:32] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1010 MB (58% inode=99%): [17:03:32] / on wolfsbane is OK: DISK OK - free space: / 16520 MB (55% inode=93%): [17:03:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:04:01] Environment IPMI on wolfsbane is OK: ok: temperature ok fan ok voltage ok chassis ok [17:21:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23383.000000 [17:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44042 [17:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44043.000000 [17:28:32] Free Memory on damiana is WARNING: WARNING - 6.1% (509872 kB) free! [17:33:32] Free Memory on damiana is OK: OK - 7.2% (602516 kB) free. [17:33:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:36:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:39:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:39:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:40:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:40:32] Free Memory on damiana is WARNING: WARNING - 6.2% (520360 kB) free! [17:40:33] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:40:42] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:40:42] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:40:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:43:07] Platonides * Re: [Toolserver-l] Why does Perl use so much more vmem on Linux than on Solaris? [17:48:32] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 11242 MB (0% inode=94%): [17:48:41] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:32] SMTP on hemlock is OK: SMTP OK - 0.016 sec. response time [17:53:32] / on hemlock is OK: DISK OK - free space: / 7008 MB (35% inode=89%): [17:53:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48315 MB (7% inode=99%): [17:53:32] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [17:53:32] Load avg. on hemlock is OK: OK - load average: 0.37, 0.26, 0.24 [17:53:33] /aux0 on hemlock is OK: DISK OK - free space: /aux0 733640 MB (13% inode=53%): [17:53:33] /home on hemlock is OK: DISK OK - free space: /home 21280 MB (42% inode=88%): [17:53:41] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 81 MB (14% inode=98%): [17:53:51] CAM on hemlock is OK: OK - cam detected no new errors [17:53:51] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [17:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19954.000000 [18:03:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:06:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:09:32] Free Memory on damiana is WARNING: WARNING - 7.0% (585196 kB) free! [18:12:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:12:32] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:12:42] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:12:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:12:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:13:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:13:41] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:32] Free Memory on damiana is OK: OK - 8.2% (691128 kB) free. [18:15:07] @replag [18:15:08] liangent: s1-rr-a: 12h 2m 55s [-0.22 s/s]; s1-user: 12h 2m 55s [-0.22 s/s]; s2-user: 15h 3m 15s [+0.55 s/s]; s2-user-c: 5h 36m 4s [+0.50 s/s]; s3-rr-a: 20s [-0.02 s/s]; s3-user: 20s [-0.02 s/s]; s5-user-c: 5h 36m 5s [+0.50 s/s] [18:21:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23252.000000 [18:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43160 [18:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43156.000000 [18:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48121 MB (7% inode=99%): [18:57:03] @replag [18:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:57:51] CAM on hemlock is OK: OK - cam detected no new errors [18:57:51] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [18:58:32] / on hemlock is OK: DISK OK - free space: / 7006 MB (35% inode=89%): [18:58:32] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:58:32] Load avg. on hemlock is OK: OK - load average: 0.23, 0.20, 0.21 [18:58:32] /home on hemlock is OK: DISK OK - free space: /home 21261 MB (42% inode=88%): [18:58:32] /aux0 on hemlock is OK: DISK OK - free space: /aux0 733015 MB (13% inode=53%): [18:58:42] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 76 MB (14% inode=98%): [18:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:59:27] @replag [18:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21702.000000 [19:00:09] where's tsbot? [19:00:48] * Brooke kicks tsbot. [19:01:32] Free Memory on damiana is WARNING: WARNING - 6.2% (517048 kB) free! [19:03:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:04:32] Free Memory on damiana is OK: OK - 7.3% (609152 kB) free. [19:05:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:11:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:11:32] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:11:41] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:11:41] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:11:41] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:11:51] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:15:32] / on hemlock is CRITICAL: NRPE: Unable to read output [19:15:33] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:15:33] /aux0 on hemlock is CRITICAL: Connection refused by host [19:15:33] /home on hemlock is CRITICAL: Connection refused by host [19:15:33] Load avg. on hemlock is CRITICAL: Connection refused by host [19:15:41] /tmp on hemlock is CRITICAL: Connection refused by host [19:15:52] Environment IPMI on hemlock is CRITICAL: Connection refused by host [19:15:52] CAM on hemlock is CRITICAL: Connection refused by host [19:20:34] @replag [19:20:35] liangent: s1-rr-a: 11h 52m 28s [+0.78 s/s]; s1-user: 11h 52m 28s [+0.78 s/s]; s2-user: 15h 36m 38s [+0.51 s/s]; s2-user-c: 6h 9m 56s [+0.24 s/s]; s3-rr-a: 60s [-0.79 s/s]; s3-user: 60s [-0.79 s/s]; s4-user: 5m 43s [+1.00 s/s]; s5-user-c: 6h 9m 57s [+0.25 s/s] [19:21:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23871.000000 [19:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42668 [19:26:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42675.000000 [19:28:32] Free Memory on damiana is WARNING: WARNING - 6.4% (539692 kB) free! [19:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:34:42] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:35:32] Free Memory on damiana is OK: OK - 8.9% (743248 kB) free. [19:35:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47875 MB (7% inode=99%): [19:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:59:41] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 78 MB (18% inode=98%): [19:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23087.000000 [20:09:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:11:42] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:12:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:14:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:14:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:15:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:15:32] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:15:41] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:15:41] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:15:41] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25264.000000 [20:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41859 [20:26:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41858.000000 [20:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:36:07] DaB. * Re: [Toolserver-l] Why does Perl use so much more vmem on Linux than on Solaris? [20:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:44:39] was hoping there would be a status message somewhere [20:47:13] UTRS has been down for ages, database error (suspect timeout) [20:54:31] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47508 MB (7% inode=99%): [20:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25594.000000 [21:08:32] Free Memory on damiana is WARNING: WARNING - 6.8% (569620 kB) free! [21:09:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:12:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:14:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:14:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:15:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:15:41] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:15:41] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:15:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:42] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:18:32] Free Memory on damiana is OK: OK - 7.9% (664868 kB) free. [21:20:41] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27096.000000 [21:25:41] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [21:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40669 [21:26:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40657.000000 [21:26:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:28:32] Free Memory on damiana is WARNING: WARNING - 5.4% (452716 kB) free! [21:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:35:32] Free Memory on damiana is OK: OK - 8.2% (683488 kB) free. [21:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47350 MB (7% inode=99%): [21:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:59:31] Free Memory on damiana is WARNING: WARNING - 7.0% (585984 kB) free! [21:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26453.000000 [22:00:32] Free Memory on damiana is OK: OK - 7.3% (610988 kB) free. [22:09:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:14:51] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:14:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:15:31] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:15:41] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:15:42] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:15:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:41] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:20:41] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29412.000000 [22:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40131 [22:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40131.000000 [22:27:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:30:32] SMTP on hemlock is OK: SMTP OK - 0.014 sec. response time [22:30:41] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:31:31] / on hemlock is OK: DISK OK - free space: / 6985 MB (34% inode=89%): [22:31:32] /aux0 on hemlock is OK: DISK OK - free space: /aux0 731813 MB (13% inode=53%): [22:31:33] /home on hemlock is OK: DISK OK - free space: /home 21293 MB (42% inode=88%): [22:31:42] Load avg. on hemlock is OK: OK - load average: 0.46, 0.33, 0.22 [22:31:42] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 64 MB (15% inode=98%): [22:31:52] CAM on hemlock is WARNING: NRPE: Unable to read output [22:31:52] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [22:32:51] CAM on hemlock is OK: OK - cam detected no new errors [22:33:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:35:42] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 74 MB (17% inode=98%): [22:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:40:52] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:41:21] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 31094 MB (7% inode=99%): [22:44:47] @replag [22:44:48] russblau: s1-rr-a: 11h 6m 52s [-0.22 s/s]; s1-user: 11h 6m 52s [-0.22 s/s]; s2-user: 15h 42m 30s [+0.03 s/s]; s2-user-c: 7h 16m 11s [+0.32 s/s]; s3-rr-a: 25s [-0.00 s/s]; s3-user: 25s [-0.00 s/s]; s5-user-c: 7h 16m 11s [+0.32 s/s]; s6-rr-a: 29s [+0.00 s/s] [22:44:49] russblau: s6-user: 29s [+0.00 s/s] [22:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 47192 MB (7% inode=99%): [22:55:32] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92319 MB (9% inode=98%): [22:56:31] Free Memory on damiana is WARNING: WARNING - 6.7% (559932 kB) free! [22:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:58:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:59:41] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 76 MB (17% inode=98%): [22:59:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26007.000000 [23:00:32] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419712 kB) free! [23:02:32] Free Memory on damiana is WARNING: WARNING - 5.9% (497488 kB) free! [23:05:52] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:06:02] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:06:02] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92048 MB (9% inode=98%): [23:07:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 27066 MB (6% inode=99%): [23:07:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91915 MB (9% inode=98%): [23:22:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31450.000000 [23:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39875 [23:26:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 121372 MB (12% inode=99%): [23:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39866.000000 [23:27:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:33:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:34:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:34:42] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 93 MB (20% inode=98%): [23:35:42] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:41] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:41:42] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:41:42] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:41:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:41:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:42:31] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46947 MB (7% inode=99%): [23:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27998.000000