[00:01:32] Free Memory on damiana is WARNING: WARNING - 5.4% (455200 kB) free! [00:02:31] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421060 kB) free! [00:17:32] Free Memory on damiana is WARNING: WARNING - 5.1% (426036 kB) free! [00:17:42] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:19:32] Load avg. on daphne is WARNING: WARNING - load average: 16.25, 15.63, 15.06 [00:22:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33686.000000 [00:23:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (391864 kB) free! [00:26:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40070 [00:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40069.000000 [00:27:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:29:24] Hello [00:29:40] It seems I can't login to the Toolserver phpMyAdmin [00:29:53] Is anyone else having a problem with that? [00:35:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:35:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:37:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:37:32] Free Memory on damiana is WARNING: WARNING - 5.6% (465248 kB) free! [00:41:42] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:41:42] SSH on hemlock is CRITICAL: Server answer: [00:41:42] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:41:42] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:41:52] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:41:52] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:42:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:45:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (407624 kB) free! [00:47:32] Free Memory on damiana is WARNING: WARNING - 5.1% (423876 kB) free! [00:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46549 MB (7% inode=99%): [00:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26041.000000 [01:07:32] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (351288 kB) free! [01:09:32] SMTP on hemlock is OK: SMTP OK - 0.047 sec. response time [01:10:32] Free Memory on damiana is WARNING: WARNING - 5.2% (435880 kB) free! [01:13:52] CAM on hemlock is OK: OK - cam detected no new errors [01:13:52] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [01:14:31] / on hemlock is OK: DISK OK - free space: / 6982 MB (34% inode=89%): [01:14:32] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421816 kB) free! [01:14:32] Load avg. on hemlock is OK: OK - load average: 0.21, 0.14, 0.13 [01:14:32] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:14:32] /aux0 on hemlock is OK: DISK OK - free space: /aux0 729844 MB (13% inode=53%): [01:14:41] /home on hemlock is OK: DISK OK - free space: /home 21244 MB (42% inode=88%): [01:14:41] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 93 MB (20% inode=98%): [01:15:32] Free Memory on damiana is WARNING: WARNING - 5.8% (489384 kB) free! [01:19:32] Load avg. on daphne is WARNING: WARNING - load average: 15.28, 15.29, 15.44 [01:23:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36326.000000 [01:25:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39826 [01:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39826.000000 [01:27:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:31:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:31:42] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:31:42] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:31:42] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:31:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:31:51] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [01:35:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:38:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:48:32] Load avg. on daphne is OK: OK - load average: 13.49, 14.32, 14.94 [01:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44851 MB (7% inode=99%): [01:57:31] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23124.000000 [02:00:41] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38625.000000 [02:25:41] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39820 [02:26:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39822.000000 [02:27:31] Free Memory on damiana is WARNING: WARNING - 5.6% (469544 kB) free! [02:27:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:31:32] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:31:42] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:31:42] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:31:42] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:31:42] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:51] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:31:51] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:35:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:38:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:39:31] Free Memory on damiana is OK: OK - 7.2% (603488 kB) free. [02:46:31] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (401440 kB) free! [02:54:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46128 MB (7% inode=99%): [02:57:32] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:58:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:59:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20685.000000 [03:00:41] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:52] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:03:02] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:03:02] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:03:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 30827 MB (7% inode=99%): [03:03:32] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91917 MB (9% inode=98%): [03:03:32] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91917 MB (9% inode=98%): [03:23:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41289.000000 [03:26:03] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40163 [03:26:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40163.000000 [03:28:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:31:43] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:32:03] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:32:03] SSH on hemlock is CRITICAL: Server answer: [03:32:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:32:03] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:32:03] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:32:04] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45753 MB (7% inode=99%): [03:57:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:00:02] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14369.000000 [04:00:53] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42444.000000 [04:26:03] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40561 [04:26:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40579.000000 [04:28:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:31:43] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:32:02] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:32:12] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:33:03] SSH on hemlock is CRITICAL: Server answer: [04:33:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:33:03] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:33:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [04:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:39:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24363 MB (5% inode=99%): [04:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45557 MB (7% inode=99%): [04:57:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:00:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9456.000000 [05:00:53] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:05:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25926 MB (6% inode=99%): [05:06:23] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 20709 MB (5% inode=99%): [05:18:22] @replag [05:18:23] Dispenser: s1-rr-a: 11h 21m 5s [+0.04 s/s]; s1-user: 11h 21m 5s [+0.04 s/s]; s2-user: 13h 49m 46s [-0.29 s/s]; s2-user-c: 1h 44m 45s [-0.84 s/s]; s3-rr-a: 1m 45s [+0.00 s/s]; s3-user: 1m 45s [+0.00 s/s]; s5-rr-a: 1m 13s [+0.00 s/s]; s5-user: 1m 13s [+0.00 s/s] [05:18:24] Dispenser: s5-user-c: 1h 44m 45s [-0.84 s/s] [05:24:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43884.000000 [05:26:13] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40933 [05:27:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40936.000000 [05:28:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3512.000000 [05:28:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:31:44] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:32:12] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:32:12] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:33:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:33:03] SSH on hemlock is CRITICAL: Server answer: [05:33:03] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:33:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:34:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1750.000000 [05:35:39] @replag [05:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:36:26] @replag [05:36:45] bot died again? [05:38:10] liangent: s1-rr-a: 11h 24m 22s [+0.19 s/s]; s1-user: 11h 24m 5s [+0.17 s/s]; s2-user: 13h 47m 25s [-0.13 s/s]; s2-user-c: 19m 45s [-4.69 s/s]; s3-rr-a: 1m 31s [-0.01 s/s]; s3-user: 1m 31s [-0.01 s/s]; s4-user: error; s5-user-c: 19m 40s [-4.66 s/s] [05:38:38] liangent: s6-rr-a: 49s [+0.00 s/s]; s7-rr-a: 18s [+0.00 s/s] [05:38:39] liangent: s1-rr-a: 11h 24m 49s [+0.18 s/s]; s1-user: 11h 24m 49s [+0.29 s/s]; s2-user: 13h 47m 21s [-0.03 s/s]; s2-user-c: 16m 21s [-2.05 s/s]; s4-user: 47s [-0.01 s/s]; s5-user-c: 15m 17s [-2.83 s/s]; s6-rr-a: 24s [-0.22 s/s]; s6-user: 24s [-0.00 s/s] [05:38:40] liangent: s7-rr-a: 10s [-0.09 s/s]; s7-user: 10s [+0.00 s/s] [05:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:48:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40457 MB (9% inode=99%): [05:48:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 14842 MB (1% inode=95%): [05:54:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45416 MB (7% inode=99%): [05:57:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:00:52] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:03] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:21:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46039.000000 [06:26:12] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41286 [06:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41286.000000 [06:27:03] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:28:53] toolserver.org HTTP on wolfsbane is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.984 second response time [06:30:02] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.120 second response time [06:30:53] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.008 second response time [06:31:42] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:32:13] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:32:13] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:33:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:33:12] SSH on hemlock is CRITICAL: Server answer: [06:33:13] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:33:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:35:02] /tmp on wolfsbane is CRITICAL: Connection refused by host [06:35:12] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [06:35:43] Load avg. on wolfsbane is CRITICAL: Connection refused by host [06:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:36:03] / on wolfsbane is CRITICAL: Connection refused by host [06:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:03] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [06:48:53] @replag [06:54:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45261 MB (7% inode=99%): [06:57:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:59:03] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 7.675 second response time [06:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:00:53] SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:53] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.464 second response time [07:16:53] SMTP on hemlock is OK: SMTP OK - 7.916 sec. response time [07:17:03] /home on hemlock is OK: DISK OK - free space: /home 21233 MB (42% inode=88%): [07:17:03] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:17:13] Load avg. on hemlock is OK: OK - load average: 0.11, 0.14, 0.13 [07:17:13] /aux0 on hemlock is OK: DISK OK - free space: /aux0 726465 MB (13% inode=53%): [07:17:13] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 62 MB (15% inode=98%): [07:17:13] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [07:17:13] CAM on hemlock is OK: OK - cam detected no new errors [07:17:42] / on hemlock is OK: DISK OK - free space: / 6974 MB (34% inode=89%): [07:20:13] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:21:12] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 71 MB (16% inode=98%): [07:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48629.000000 [07:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41623 [07:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41613.000000 [07:33:13] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:33:52] hi! Guten Tag! what happened to toolserver this night? [07:35:03] /tmp on wolfsbane is CRITICAL: Connection refused by host [07:35:12] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [07:35:42] Load avg. on wolfsbane is CRITICAL: Connection refused by host [07:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:36:03] / on wolfsbane is CRITICAL: Connection refused by host [07:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:43:56] help [07:44:19] tsnag: 1 [07:44:23] tsnag: help [07:44:38] !status [07:48:02] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [07:54:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45111 MB (7% inode=99%): [07:57:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:58:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:59:13] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 78 MB (18% inode=98%): [08:01:52] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.013 second response time [08:08:13] /tmp on hemlock is CRITICAL: Connection refused by host [08:09:12] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:13:13] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 62 MB (15% inode=98%): [08:15:17] @replag [08:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46853.000000 [08:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41866 [08:27:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41858.000000 [08:32:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:32:43] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:33:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:33:03] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:33:13] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:33:13] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:33:13] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:03] /tmp on wolfsbane is CRITICAL: Connection refused by host [08:35:13] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [08:35:42] Load avg. on wolfsbane is CRITICAL: Connection refused by host [08:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:36:02] / on wolfsbane is CRITICAL: Connection refused by host [08:37:12] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:41:12] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:48:03] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [08:49:13] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44931 MB (7% inode=99%): [08:57:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:58:42] / on hemlock is OK: DISK OK - free space: / 6973 MB (34% inode=89%): [08:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:59:03] /home on hemlock is OK: DISK OK - free space: /home 21296 MB (42% inode=88%): [08:59:03] Load avg. on hemlock is OK: OK - load average: 0.08, 0.12, 0.16 [08:59:03] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:59:13] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 63 MB (15% inode=98%): [08:59:13] /aux0 on hemlock is OK: DISK OK - free space: /aux0 726160 MB (13% inode=53%): [08:59:13] Environment IPMI on hemlock is WARNING: NRPE: Unable to read output [08:59:23] CAM on hemlock is OK: OK - cam detected no new errors [09:00:13] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:00:13] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:02:43] Load avg. on wolfsbane is OK: OK - load average: 1.36, 1.01, 0.88 [09:03:03] / on wolfsbane is OK: DISK OK - free space: / 17194 MB (57% inode=93%): [09:03:03] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 318 MB (72% inode=99%): [09:03:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:03:22] Environment IPMI on wolfsbane is OK: ok: temperature ok fan ok voltage ok chassis ok [09:04:12] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 70 MB (16% inode=98%): [09:04:13] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [09:11:13] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:11:42] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:12:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:12:13] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:17:02] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:18:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:18:26] @help [09:18:27] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.2.6 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [09:18:27] Nirvanchik: (help [] []) -- This command gives a useful description of what does. is only necessary if the command is in more than one plugin. [09:18:39] @commands [09:18:39] Commands: channellist, trusted, trustadd, trustdel, info, configure, infobot-link, infobot-share-trust+, infobot-share-trust-, infobot-share-off, infobot-share-on, infobot-off, refresh, infobot-on, drop, whoami, add, reload, suppress-off, suppress-on, help, RC-, recentchanges-on, language, infobot-ignore+, infobot-ignore-, recentchanges-off, logon, logoff, recentchanges-, recentchanges+, RC+ [09:20:55] @replag [09:21:03] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45451.000000 [09:24:13] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41325 [09:27:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41349.000000 [09:28:12] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:02] Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:30:13] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:34:12] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:38:13] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [09:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:39:42] Load avg. on wolfsbane is CRITICAL: Connection refused by host [09:40:03] / on wolfsbane is CRITICAL: Connection refused by host [09:40:03] /tmp on wolfsbane is CRITICAL: Connection refused by host [09:40:03] /home on hemlock is OK: DISK OK - free space: /home 21283 MB (42% inode=88%): [09:40:13] /aux0 on hemlock is OK: DISK OK - free space: /aux0 725977 MB (13% inode=53%): [09:40:43] / on hemlock is CRITICAL: Connection refused by host [09:41:02] Load avg. on hemlock is CRITICAL: Connection refused by host [09:41:12] /tmp on hemlock is CRITICAL: Connection refused by host [09:41:12] CAM on hemlock is CRITICAL: Connection refused by host [09:41:12] Environment IPMI on hemlock is CRITICAL: Connection refused by host [09:45:52] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.115 second response time [09:47:03] /home on hemlock is CRITICAL: Connection refused by host [09:47:13] /aux0 on hemlock is CRITICAL: Connection refused by host [09:47:23] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:02] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [09:53:03] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 7.700 second response time [09:54:13] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44745 MB (7% inode=99%): [09:57:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:01:12] /tmp on hemlock is OK: DISK OK - free space: /tmp 1397 MB (79% inode=98%): [10:01:13] /aux0 on hemlock is OK: DISK OK - free space: /aux0 719093 MB (13% inode=53%): [10:01:13] CAM on hemlock is OK: OK - cam detected no new errors [10:01:13] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [10:01:43] / on hemlock is OK: DISK OK - free space: / 6990 MB (35% inode=89%): [10:02:03] /home on hemlock is OK: DISK OK - free space: /home 21276 MB (42% inode=88%): [10:02:03] Load avg. on hemlock is OK: OK - load average: 1.60, 0.68, 0.37 [10:16:52] toolserver.org HTTP on wolfsbane is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.572 second response time [10:17:52] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [10:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41831.000000 [10:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41210 [10:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41240.000000 [10:32:07] emijrp * Re: [Toolserver-l] Web services are down [10:35:03] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:38:12] Environment IPMI on wolfsbane is CRITICAL: Connection refused by host [10:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:39:42] Load avg. on wolfsbane is CRITICAL: Connection refused by host [10:40:02] / on wolfsbane is CRITICAL: Connection refused by host [10:40:02] /tmp on wolfsbane is CRITICAL: Connection refused by host [10:50:03] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [10:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44508 MB (7% inode=99%): [10:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:01:42] Load avg. on wolfsbane is OK: OK - load average: 1.85, 1.15, 0.86 [11:02:02] / on wolfsbane is OK: DISK OK - free space: / 17204 MB (57% inode=93%): [11:02:03] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 3339 MB (96% inode=99%): [11:02:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:02:13] Environment IPMI on wolfsbane is OK: ok: temperature ok fan ok voltage ok chassis ok [11:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38376.000000 [11:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43055 [11:26:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 121040 MB (12% inode=99%): [11:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43068.000000 [11:28:57] @replag [11:29:47] @info [11:29:47] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-toolserver.htm [11:30:23] @help [11:30:23] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.8.2.6 source code licensed under GPL and located at https://github.com/benapetr/wikimedia-bot [11:30:37] @commands [11:30:37] Commands: channellist, trusted, trustadd, trustdel, info, configure, infobot-link, infobot-share-trust+, infobot-share-trust-, infobot-share-off, infobot-share-on, infobot-off, refresh, infobot-on, drop, whoami, add, reload, suppress-off, suppress-on, help, RC-, recentchanges-on, language, infobot-ignore+, infobot-ignore-, recentchanges-off, logon, logoff, recentchanges-, recentchanges+, RC+ [11:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:54:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44271 MB (7% inode=99%): [11:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:02:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:08:42] Free Memory on damiana is WARNING: WARNING - 6.9% (577616 kB) free! [12:12:43] Free Memory on damiana is OK: OK - 7.3% (610324 kB) free. [12:23:43] Free Memory on damiana is WARNING: WARNING - 6.0% (505204 kB) free! [12:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36378.000000 [12:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44916 [12:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44932.000000 [12:31:19] hello all [12:31:33] I will reboot wolfsbane to solve the /tmp-problem [12:32:23] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24392 MB (5% inode=99%): [12:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:36:03] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [12:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:40:43] Free Memory on damiana is OK: OK - 7.1% (593872 kB) free. [12:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:51:12] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 86 MB (19% inode=98%): [12:54:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44000 MB (7% inode=99%): [12:58:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 26280 MB (6% inode=99%): [12:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:58:43] Free Memory on damiana is WARNING: WARNING - 6.5% (546076 kB) free! [12:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:59:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:59:23] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 20644 MB (5% inode=99%): [13:06:13] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:09:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:12:03] /home on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:12:12] Load avg. on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:12:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:12:13] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:12:13] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:12:22] SSH on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:42] / on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26814.000000 [13:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47348 [13:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47367.000000 [13:28:42] Free Memory on damiana is OK: OK - 7.4% (616640 kB) free. [13:31:03] /home on hemlock is OK: DISK OK - free space: /home 21314 MB (42% inode=88%): [13:31:03] Load avg. on hemlock is OK: OK - load average: 0.63, 0.45, 0.38 [13:31:07] DaB. * Re: [Toolserver-l] Web services are down [13:31:13] SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:31:13] /tmp on hemlock is WARNING: DISK WARNING - free space: /tmp 61 MB (14% inode=98%): [13:31:43] / on hemlock is CRITICAL: Connection refused by host [13:32:04] @replag [13:32:12] /aux0 on hemlock is CRITICAL: Connection refused by host [13:32:13] CAM on hemlock is CRITICAL: Connection refused by host [13:32:13] /tmp on hemlock is CRITICAL: Connection refused by host [13:32:13] Environment IPMI on hemlock is CRITICAL: Connection refused by host [13:32:28] * Dispenser kicks tsbot [13:32:43] / on hemlock is OK: DISK OK - free space: / 6982 MB (34% inode=89%): [13:32:58] @replag [13:33:13] /aux0 on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:33:13] CAM on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:33:13] /tmp on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:33:13] Environment IPMI on hemlock is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [13:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:36:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42104 MB (10% inode=99%): [13:37:12] /aux0 on hemlock is OK: DISK OK - free space: /aux0 715644 MB (13% inode=52%): [13:37:12] CAM on hemlock is OK: OK - cam detected no new errors [13:37:12] Environment IPMI on hemlock is OK: ok: temperature ok fan ok voltage ok chassis ok [13:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:47:46] @replag [13:48:23] russblau: s1-rr-a: 13h 22m 35s [+0.59 s/s]; s1-user: 13h 22m 35s [+0.59 s/s]; s2-user: 5h 31m 51s [-0.05 s/s]; s3-rr-a: 2m 51s [+0.22 s/s]; s3-user: 2m 51s [+0.22 s/s]; s6-rr-a: 4m 22s [+0.47 s/s]; s6-user: 4m 22s [+0.47 s/s]; s7-rr-a: 12m 43s [+0.97 s/s] [13:48:24] russblau: s7-user: 12m 53s [+0.97 s/s] [13:54:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43749 MB (7% inode=99%): [13:58:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:03:23] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23600 MB (5% inode=99%): [14:04:43] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 32.98, 24.47, 15.05 [14:07:43] Load avg. on ortelius is WARNING: WARNING - load average: 23.27, 24.61, 16.89 [14:09:43] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 27.79, 25.55, 18.19 [14:16:12] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2037 [14:22:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35820 MB (8% inode=99%): [14:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14905.000000 [14:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49506 [14:27:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49521.000000 [14:31:43] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 18.84, 19.84, 21.18 [14:33:22] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24243 MB (5% inode=99%): [14:35:43] Load avg. on ortelius is WARNING: WARNING - load average: 15.05, 17.10, 19.71 [14:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:44:13] MySQL slave on z-dat-s7-a is OK: Uptime: 8765328 Threads: 15 Questions: 2328055125 Slow queries: 307015 Opens: 20029202 Flush tables: 1 Open tables: 7090 Queries per second avg: 265.598 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1738 [14:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:53:42] Load avg. on ortelius is OK: OK - load average: 7.53, 10.82, 14.68 [14:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43417 MB (7% inode=99%): [14:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:03:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92537 MB (9% inode=98%): [15:03:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92537 MB (9% inode=98%): [15:07:43] Free Memory on damiana is WARNING: WARNING - 6.4% (532200 kB) free! [15:12:43] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (390968 kB) free! [15:14:43] Free Memory on damiana is WARNING: WARNING - 6.0% (500372 kB) free! [15:19:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 32998 MB (8% inode=99%): [15:24:03] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7402.000000 [15:25:42] Free Memory on damiana is OK: OK - 7.4% (621592 kB) free. [15:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50885 [15:27:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50896.000000 [15:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:42:02] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3560.000000 [15:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:52:03] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1744.000000 [15:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43148 MB (7% inode=99%): [15:58:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:08:43] Free Memory on damiana is WARNING: WARNING - 6.7% (560804 kB) free! [16:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50176 [16:26:42] Free Memory on damiana is OK: OK - 7.1% (598732 kB) free. [16:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50089.000000 [16:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:46:44] Free Memory on damiana is WARNING: WARNING - 7.0% (586508 kB) free! [16:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42915 MB (7% inode=99%): [16:57:43] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (363936 kB) free! [16:58:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:15:23] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23790 MB (5% inode=99%): [17:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45542 [17:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45498.000000 [17:35:43] Free Memory on damiana is WARNING: WARNING - 6.2% (517664 kB) free! [17:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:48:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 16906 MB (1% inode=96%): [17:53:42] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (336880 kB) free! [17:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41875 MB (6% inode=99%): [17:55:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:12] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:55:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92265 MB (9% inode=98%): [17:55:53] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92265 MB (9% inode=98%): [17:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:00:22] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38968 MB (9% inode=99%): [18:10:43] Free Memory on damiana is WARNING: WARNING - 6.0% (500356 kB) free! [18:26:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37787 [18:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37721.000000 [18:30:43] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (402592 kB) free! [18:34:43] Free Memory on damiana is WARNING: WARNING - 5.1% (431164 kB) free! [18:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:44:43] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (397932 kB) free! [18:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41585 MB (6% inode=99%): [18:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:26:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33041 [19:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33030.000000 [19:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:45:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41245 MB (6% inode=99%): [19:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:25:42] Free Memory on damiana is WARNING: WARNING - 6.3% (531908 kB) free! [20:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27144 [20:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27092.000000 [20:29:42] Free Memory on damiana is OK: OK - 8.0% (674304 kB) free. [20:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:48:51] @replag [20:48:51] multichill: s1-rr-a: 7h 6m 10s [-0.98 s/s]; s1-user: 7h 6m 10s [-0.98 s/s]; s2-user: 1m 40s [-0.78 s/s]; s3-rr-a: 16s [-0.04 s/s]; s3-user: 16s [-0.04 s/s]; s5-rr-a: 2m 48s [-]; s5-user: 2m 48s [-] [20:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40996 MB (6% inode=99%): [20:56:05] DaBPunkt, ssh_keysign doesn't seem to be working from willow [20:56:17] (try to jump from there to another server) [20:58:37] Platonides: a server of us or a external one? [20:58:42] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:59:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:00:09] Platonides: "mzmcbride@willow$ ssh nightshade" seems to work fine for me. [21:03:02] Brooke, $ ssh nightshade [21:03:02] no matching hostkey found [21:03:03] ssh_keysign: no reply [21:03:03] key_sign failed [21:03:03] Password: [21:03:16] forget me [21:03:33] I didn't notice I was in a project account [21:03:40] :-) [21:03:46] ID10T error :) [21:15:07] Any roots here that can help me? [21:15:17] I'm running into an issue with file permissions in an MMP home directory [21:15:47] DaB. is probably around. [21:15:49] someone who is no longer active in the project modified some files at some point that are chgrp'ed "users" instead of our group, and chmod doesn't allow me to set it either (755/644) [21:16:19] /home/project/c/v/n/cvn/bots/SentryBot [21:16:42] currently chmod 755 misza13 users; should be 775 misza13 cvn (or while at it, cvn cvn) [21:17:01] chgrp cvn, and chmod 775 will do [21:17:16] (recursive for chrp) [21:17:31] and for files 644 recursive, 755 for directories. [21:17:33] 775 * [21:17:41] basically the sane default for MMPs [21:17:43] Free Memory on damiana is WARNING: WARNING - 6.1% (508036 kB) free! [21:17:56] DaBPunkt: :) [21:18:23] changed the group. You should be able to set the chmod yourself [21:18:46] DaBPunkt: Is there a way to make this the default for anything in that home directory ? (chmod 775/664 and chgrp cvn) [21:19:04] I am really getting old of this, nearly ever other week I get stuck on this stuff [21:19:24] both my own files (with colleguages) and files from others [21:19:26] Thx [21:20:33] you can stick the group (+s), but it is not recursive – you have to repeat it for each directory [21:20:42] Free Memory on damiana is OK: OK - 7.4% (616536 kB) free. [21:21:37] DaBPunkt: If I "stick the group" on the home dir, will that make new dirs inside also sticky ? [21:21:47] (which is the desired behavior) then I only have to make a one-time sweep [21:23:04] yes, but only new ones [21:23:08] Krinkle: mv evilfile evilfile.old && cp evilfile.old evilfile && rm evilfile [21:23:32] felicity: I know, that's cute for one file.. [21:23:45] Or perhaps something with umask? I mean.. If I create files in my own home directory they are automatically creates as "644 krinkle users" (not krinkle krinkle). Which means something is making the ts-users group default (which is nice). Perhaps that can be done in the cvn home dir, but for the cvn group ? [21:24:04] (file-rights are one of the few right that are better solved in windows IMHO) [21:24:26] DaBPunkt: or ZFS. this is easy to fix there. (because it uses Windows ACLs) [21:25:04] DaBPunkt: btw, seems I can't chmod it, it says I have to be the file owner [21:25:05] cvn@willow:~/bots/SentryBot$ chmod 770 Config/IRCConsole.conf [21:25:06] chmod: changing permissions of `Config/IRCConsole.conf': Not owner [21:25:08] -rwxrwxrwx 1 misza13 cvn 752 Feb 6 2011 IRCConsole.conf [21:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23873 [21:26:34] Krinkle: files you create are owned by your primary group. you can't change that [21:26:46] ok [21:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23841.000000 [21:27:51] I just want to have simple project without having to worry about permissions, I understand how unix works a bit but can't imagine there is no way to have this work properly [21:28:08] can the copy-trick be applied to a directory ? [21:30:16] Here's the main problem right now: http://paste2.org/p/2186697 [21:31:12] it would be awesome if that could get fixed, but I'm afraid it will be messed up within a week again. [21:32:47] DaBPunkt: So the sticky thing, is that "$ chmod g+s dirname" ? [21:33:01] yes [21:35:11] and then there is directories that are in another state of stuck such as /home/(...)/cvn/local it is owned by me, and grouped 'cvn', but nor cvn nor me can re-chown it. From both ends it says, not owner [21:35:18] I want to chown it 'cvn' [21:35:36] I'd think I can do that for files I own, especially for groups I'm member of [21:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:36:18] Krinkle: on generall: Working together as a group in linux is a bit tricky and you have to use discipline – but as long as you can speak with the owner and the group is writeable it is not that bad [21:37:03] yeah, the files don't have to be owned by cvn, group=cvn, and group-writable is enough. But that's not always the case. [21:37:11] and apparently sticky-group can only be set when the file is owned [21:37:14] dir* [21:37:42] stciky can only be set on dirs [21:37:46] Yes [21:37:56] but I can;'t set it on /home/(..)/cvn/local [21:38:07] (there is also +s for files – but that's another thing) [21:38:14] because cvn doesn't own it, but krinkle. and from krinkle it also says "not owner" [21:38:34] it is group writable and group is cvn [21:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:38:58] I could copy it and re-create under the name 'local'. So it seems silly it doesn't allow it. [21:42:00] DaBPunkt: the local sir is pretty deep, I don't want to copy every single file over just to be able to write it. Am I missing something here or is this just a stupid flaw in how unix bitches about everything? [21:43:29] Krinkle: try "rsync -r " [21:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:47:03] k [21:50:37] DaBPunkt: nice [21:51:00] DaBPunkt: btw, any idea what ".nfs65A30C2" etc. are? I see these files popping up now, when I do rsync and try to delete the "wrong" one [21:51:09] If I remove them a new one takes it place instantly [21:51:14] (with a slightly different name) [21:51:22] you can't remove them – that's the reason :) [21:51:40] that are nfs-lock-files. [21:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40693 MB (6% inode=99%): [21:55:04] k, I've move those problem dirs (which are other wise, except for the nfs lock files) and moved them into ~/tmp/ that seems to work [21:55:05] I can rename the dir, just not delete it [21:55:05] will figure out later [21:55:37] which dir? [21:56:58] stuff inside /home/(..)/cvn/tmp [21:57:00] (two dirs) [21:57:43] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (331116 kB) free! [21:57:44] deleted [21:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:58:55] thx [21:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:59:21] DaBPunkt: I'll run the sticky group later on the whole home, hoping that'll fix it for the future [21:59:35] (currently still rsync-ing some dirs) [22:00:09] if you like I can set the group for one (1 time service of course ;)) [22:00:14] one → you [22:01:01] that would be nice. 775/664 and chown/chgrp cvn? [22:01:24] (or just chown/chgrp, I suppose I can change chmod later) [22:01:43] Free Memory on damiana is OK: OK - 7.5% (627140 kB) free. [22:03:11] ok, should be done [22:03:18] DaBPunkt: while you're here, i've got one other thing I was hoping you have an idea on. As you may know, the cvn has a secure web-interface written in PHP for non-programmer staff members (that don't have ts accounts) to start/stop our irc bots in #cvn-sw, #cvn-commons etc. [22:03:21] Right now it is very hacky, there is a fixed set of shell commands (only editable by programmers in a php file) with symbolic names that they can use to run. However since I don't want to run them on the web servers, I put them in a mysql table as queue and then on willow a cron that polls that db table and executes them. I'd much rather have those commands somehow be executed directly, but haven't figured out a way to do so. [22:05:03] somehow ssh into willow/nightshade (or submit for that matter) and execute a command - from php running on the web server. [22:05:04] I noticed that within toolserver, users can ssh into other machines without needing to forward keys, that's nice I hope that is "key" to being able to accomplish this [22:06:36] sorry, that's sounds way to dangerous that I would like to help. [22:07:15] I my eyes a limited command-set (like you have now) is the better way than allowing people to run commands [22:07:24] Oh now, nothing like that [22:07:34] it will still me just as limited [22:07:41] oh god, I phrased it terrible [22:08:42] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (393756 kB) free! [22:08:46] right now they submit a html form and submit the symbolic name of the command, the php process then adds that name to the queue, and on willow it looks up the command and executes it. But willow does so from a cron, polling the mysql table (meaning a one minute delay, though it could be a long running php process in a while loop) [22:09:11] what I'd like to do instead is have the php submission handler, look up the command, ssh into willow, nightshade (or submit) and execute it there. [22:10:00] or even for better security, ssh into there and execute a wrapper script passing the name as an argument. [22:10:16] you have a problem because an IRC needs a minute or 2 to start? [22:10:31] IRC → IRC-bot [22:10:59] it starts in seconds, the problem is that it is a bit complex right now when working with a team. [22:11:20] because they see the bot is off, go to the control panel to start it and then there's two, because someone else (within the minute it took for cron) already did it [22:11:41] and that's the ideal scenario, in practice it turns out that for some reason the cron isn't running most of the time [22:12:27] I've tried to figure this out time and time again, but for some reason it just won't run every minute (it only takes like 2 seconds to run) [22:12:42] cvn's crontab [22:12:54] * * * * * php $HOME/backend/cronjob_cvncp.php > $HOME/backend/cronjob_cvncp.log 2>&1 [22:13:11] put the "commands" in a mysql-table, create a background-programm that checks the mysql-table from time to time, use SGE to prevent double-starts [22:13:18] Last run: (842 seconds ago) [22:13:31] we already use SGE :) [22:13:40] Oh, but not for that. [22:13:49] Great [22:15:07] DaBPunkt: would it be okay to run that background program from submit ? [22:15:42] And then add an SGE cronsub in cronie there as well to ensure that that background program is always running. [22:16:08] of course [22:16:16] alrighty [22:16:24] (but do not start it EVERY minute…) [22:17:02] alrighty [22:17:08] every other minute ;-) [22:17:12] nah, 15 minutes is okay [22:18:16] the cvn channel users are pretty sensitive to downtime though, because in practice/reality it is simply the case that when the cvn bots are off, edits made then are not going to be reviewed. [22:18:23] at least not with those eyes [22:19:53] I'm pretty sensitive to extravagance… [22:21:12] I agree. I'm just maintaining the network, I haven't been patrolling for months. Too much other stuff to do. [22:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20677 [22:27:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20621.000000 [22:27:17] @replag [22:27:18] DaBPunkt: s1-rr-a: 5h 42m 54s [-0.85 s/s]; s1-user: 5h 42m 54s [-0.85 s/s] [22:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:38:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:45:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:49:47] nacht ts [22:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40164 MB (6% inode=99%): [22:58:44] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:26:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18358 [23:26:42] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 122238 MB (12% inode=99%): [23:27:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18321.000000 [23:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:38:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:45:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:54:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39840 MB (6% inode=99%): [23:58:43] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:59:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default