[00:01:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1012998 [00:09:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 419573 MB (7% inode=40%): [00:11:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:13:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:15:13] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 98945 MB (10% inode=99%): [00:23:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:25:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1013791.000000 [00:33:03] Free Memory on turnera is WARNING: WARNING - 6.9% (575832 kB) free! [00:38:03] Free Memory on turnera is OK: OK - 7.2% (600100 kB) free. [00:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:40:38] @replag [00:40:39] Brooke: s1-user: 1w 4d 17h 44m 56s [+0.45 s/s]; s2-user: 19s [-0.00 s/s]; s3-rr-a: 24s [-0.00 s/s]; s3-user: 24s [-0.00 s/s]; s6-rr-a: 60s [-0.00 s/s]; s6-user: 60s [-0.00 s/s] [01:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1015006 [01:09:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 419346 MB (7% inode=40%): [01:11:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:14:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:23:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:25:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1015483.000000 [01:38:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:40:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 95905 MB (9% inode=98%): [01:42:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 95902 MB (9% inode=98%): [01:54:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1016305 [02:09:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 419170 MB (7% inode=40%): [02:11:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:14:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:23:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:25:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1016903.000000 [02:32:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96134 MB (9% inode=98%): [02:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1017696 [03:09:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416596 MB (7% inode=40%): [03:12:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:14:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:23:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:25:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1018216.000000 [03:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:02:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1019279 [04:07:03] Free Memory on turnera is WARNING: WARNING - 6.3% (528260 kB) free! [04:09:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416877 MB (7% inode=40%): [04:10:03] Free Memory on turnera is OK: OK - 7.1% (594264 kB) free. [04:12:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:14:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:23:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:25:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1019984.000000 [04:33:33] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:38:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1021048 [05:08:34] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:09:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 417418 MB (7% inode=40%): [05:12:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:15:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:24:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:25:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1021877.000000 [05:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:45:23] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 34111 MB (2% inode=98%): [06:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1023015 [06:05:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:05:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:09:23] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 97024 MB (9% inode=98%): [06:09:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 418043 MB (7% inode=40%): [06:10:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 97022 MB (9% inode=98%): [06:12:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:15:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:24:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:26:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1023723.000000 [06:31:54] APT on yarrow is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [06:31:54] APT on nightshade is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [06:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:42:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:42:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:46:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96953 MB (9% inode=98%): [06:47:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96945 MB (9% inode=98%): [06:53:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:53:33] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1024859 [07:09:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 417840 MB (7% inode=40%): [07:12:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:12:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96944 MB (9% inode=98%): [07:13:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96944 MB (9% inode=98%): [07:15:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:24:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:27:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1025569.000000 [07:31:54] APT on nightshade is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [07:32:43] APT on yarrow is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [07:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:53:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:53:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:54:04] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96565 MB (9% inode=98%): [07:54:04] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96565 MB (9% inode=98%): [08:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1026217 [08:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 417175 MB (7% inode=40%): [08:12:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:15:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:25:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:27:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1026704.000000 [08:31:54] APT on nightshade is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [08:32:53] APT on yarrow is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [08:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:45:03] Free Memory on turnera is WARNING: WARNING - 6.5% (541020 kB) free! [08:49:04] Free Memory on turnera is OK: OK - 7.4% (619232 kB) free. [08:54:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:57:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96558 MB (9% inode=98%): [08:58:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96584 MB (9% inode=98%): [09:02:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1027350 [09:10:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414678 MB (7% inode=39%): [09:12:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:15:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:26:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:28:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1028025.000000 [09:31:53] APT on nightshade is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [09:32:54] APT on yarrow is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [09:37:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41614 MB (10% inode=99%): [09:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:02:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1028502 [10:10:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:10:23] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414410 MB (7% inode=39%): [10:10:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:10:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 96078 MB (9% inode=98%): [10:10:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 26134 MB (6% inode=99%): [10:11:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 96078 MB (9% inode=98%): [10:12:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:15:43] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:43] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:43] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:03] Free Memory on turnera is WARNING: WARNING - 6.9% (575260 kB) free! [10:16:43] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:17:34] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:17:34] SMTP on z-dat-s4-a is OK: SMTP OK - 0.005 sec. response time [10:17:34] SMTP on z-dat-s7-a is OK: SMTP OK - 0.023 sec. response time [10:17:34] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:19:04] Free Memory on turnera is OK: OK - 7.1% (593408 kB) free. [10:19:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23712 MB (5% inode=99%): [10:27:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:28:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1029054.000000 [10:31:54] APT on nightshade is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [10:32:53] APT on yarrow is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [10:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:38:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49619 MB (12% inode=99%): [10:56:24] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:56:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:56:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 94667 MB (9% inode=98%): [10:57:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 94667 MB (9% inode=98%): [11:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1030103 [11:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413897 MB (7% inode=39%): [11:12:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:16:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:27:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1031081.000000 [11:32:54] APT on nightshade is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [11:33:43] APT on yarrow is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [11:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:02:44] hello all [12:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1032176 [12:10:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413323 MB (7% inode=39%): [12:13:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:13:34] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:14:53] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 93338 MB (9% inode=98%): [12:15:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 93337 MB (9% inode=98%): [12:15:13] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 98980 MB (10% inode=99%): [12:16:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:27:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1032962.000000 [12:33:53] APT on nightshade is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [12:34:53] APT on yarrow is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [12:37:43] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:37:53] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:38:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:48:04] Free Memory on turnera is WARNING: WARNING - 6.4% (539072 kB) free! [13:03:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1033785 [13:07:04] Free Memory on turnera is OK: OK - 7.1% (595260 kB) free. [13:10:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413103 MB (7% inode=39%): [13:13:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:16:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:27:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1034471.000000 [13:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:44:03] Free Memory on turnera is WARNING: WARNING - 6.5% (546500 kB) free! [13:48:03] Free Memory on turnera is OK: OK - 7.1% (597852 kB) free. [14:03:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1035285 [14:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 412828 MB (7% inode=39%): [14:13:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:16:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:27:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:28:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1035995.000000 [14:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:53:23] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:55:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 84930 MB (8% inode=98%): [15:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1036831 [15:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 412572 MB (7% inode=39%): [15:13:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:17:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:27:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1037412.000000 [15:38:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:39:03] Free Memory on turnera is WARNING: WARNING - 6.1% (512268 kB) free! [15:50:03] Free Memory on turnera is OK: OK - 7.2% (603844 kB) free. [16:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1038227 [16:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 412276 MB (7% inode=39%): [16:13:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:17:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1038859.000000 [16:28:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:38:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:03:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1039590 [17:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 411936 MB (7% inode=39%): [17:13:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:17:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:21:32] Danny_B|backup, [17:21:51] yes? [17:22:30] In request TS-1454, [17:22:35] https://jira.toolserver.org/browse/TS-1454 [17:22:57] you said 'Please add the following sentence to your user-page: "I wish to change my SSH-key on the toolserver. See request TS-1454". And link the diff here.' [17:23:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:48] devunt: that was obviously DaBPunkt and not me ;-) [17:25:00] oh, [17:25:26] sorry :p [17:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1040244.000000 [17:28:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:38:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:43:33] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:45:23] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 38141 MB (3% inode=98%): [17:55:03] Free Memory on turnera is WARNING: WARNING - 6.7% (560248 kB) free! [18:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1041137 [18:08:04] Free Memory on turnera is OK: OK - 7.2% (604436 kB) free. [18:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 411636 MB (7% inode=39%): [18:13:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:17:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1041895.000000 [18:28:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1042875 [19:10:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414554 MB (7% inode=39%): [19:13:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:14:39] valhallasw: hey [19:17:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:18:31] liangent: hey [19:18:37] really weird issue you're having [19:18:45] I also cannot reproduce it with your os.environ [19:19:21] python -c "import sys; print sys.version" [19:19:26] ^ could you try running that? [19:19:30] it's my last guess [19:20:02] or possibly php mangling the library include path, but then I would expect to have the same issues... [19:20:31] 2.6.6 (r266:84292, Jul 11 2012, 14:46:24) [GCC 4.4.5] [19:20:48] I'm cloning a new mediawiki and see [19:20:54] ! [19:21:05] what? [19:21:07] oh, wait [19:21:11] I'm on wolfsbane [19:21:15] that's why I have 2.7 [19:21:16] argh [19:21:37] it works fine on wolfsbane iirc [19:21:50] what I mean - do you also get that when running through php? [19:22:03] or do you also have issues when you run python in the shell directly? [19:22:42] on nightshade: I'm having issues running through php. no issue directly [19:24:12] on wolfsbane: both ok [19:24:58] and you get the same python version with and without php? [19:26:32] yes, as determined by python --version [19:28:06] with a brand new mw clone and an empty LocalSettings.php: still fails [19:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1043548.000000 [19:29:19] I was running it in an old phase3 I had lying around [19:29:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:29:54] hmm do you have a new one? [19:30:15] not at the moment [19:30:21] can I clone your git repos? might be faster [19:30:47] or do you have the clone command nearby? [19:31:00] I made /tmp/core.tgz on nightshade [19:32:07] thanks [19:35:35] yes, breaks for me too, now [19:36:49] maybe I can bisect to find the breaking revision [19:37:44] what's your known-good revision? [19:38:04] Free Memory on turnera is WARNING: WARNING - 5.8% (484648 kB) free! [19:38:27] oof. [19:38:37] ? [19:38:45] 1.18wmf1 98239 [19:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:47:04] Free Memory on turnera is OK: OK - 7.3% (608064 kB) free. [19:50:52] valhalla1w: seems 1.18wmf1 is not in git so I checked out REL1_18 [19:50:56] but it still fail for me [19:51:07] REL1_17 fails too [19:51:22] it becomes stranger and stranger... [19:52:57] so can you checkout REL1_18 from git in your home and see? [19:53:22] $ git checkout remotes/origin/REL1_18 [19:53:23] $ echo "echo wfShellExec(\"python -c \\\"import locale, os; print os.environ, locale.setlocale(locale.LC_ALL, '')\\\"\");" | php maintenance/eval.php [19:53:33] I'm using these commands [19:54:32] just a sec [19:55:06] errors here too [19:56:19] really strange this... [19:58:25] ...so any idea about the reason? [19:58:26] running the locale command from php also works [19:59:16] is your localsettings.php configured somehow [20:02:05] it's the same with an empty localsettings.php [20:02:13] I'm confused now :/ [20:02:32] running the very basic 'setlocale(LC_ALL, "")' in C under PHP also works [20:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1044150 [20:06:04] I'm checkout 1.18wmf1 from svn... [20:06:50] maybe you have access to /home/valhallasw/src/phase3, too [20:11:18] hmm your version is working [20:11:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414345 MB (7% inode=39%): [20:13:28] BUT a newly checked out 1.18wmf1 from svn doesn't work [20:14:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:17:04] Free Memory on turnera is WARNING: WARNING - 6.4% (538128 kB) free! [20:17:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:18:49] liangent: so there is something strange with my version [20:19:27] I'm doing svn update -r98239 [20:20:03] Free Memory on turnera is OK: OK - 7.4% (622572 kB) free. [20:21:55] r98239 files from svn fails! [20:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1044776.000000 [20:30:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:02:03] Free Memory on turnera is WARNING: WARNING - 6.3% (531800 kB) free! [21:03:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1045546 [21:07:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42150 MB (10% inode=99%): [21:09:03] Free Memory on turnera is OK: OK - 7.3% (609560 kB) free. [21:12:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414006 MB (7% inode=39%): [21:15:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:17:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1046120.000000 [21:29:03] Free Memory on turnera is WARNING: WARNING - 6.4% (539988 kB) free! [21:30:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:32:03] Free Memory on turnera is OK: OK - 7.3% (613348 kB) free. [21:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:50:53] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23914 MB (5% inode=99%): [22:03:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1046779 [22:13:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413608 MB (7% inode=39%): [22:15:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:15:54] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 47857 MB (11% inode=99%): [22:18:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:28:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1047267.000000 [22:31:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:59:11] @replag [22:59:11] Dispenser: s1-user: 1w 5d 3h 4m 42s [+0.42 s/s]; s2-user: 48s [+0.00 s/s]; s3-rr-a: 39s [+0.00 s/s]; s3-user: 39s [+0.00 s/s]; s5-rr-a: 12s [-0.00 s/s]; s5-user: 12s [-0.00 s/s] [23:04:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1048036 [23:13:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 412904 MB (7% inode=39%): [23:16:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:19:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:28:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1048647.000000 [23:31:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:39:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:55:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36146 MB (8% inode=99%):