[00:06:26] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:10:26] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.005371/1.8, alarm hl:np_load_avg=1.203613/2.3, alarm hl:mem_free=282.000000M/300M, alarm hl:available=1/0 [00:11:45] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:12:44] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:13:17] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:14:26] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:16:45] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:19:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97445.000000 [00:20:26] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.744629/1.8, alarm hl:np_load_avg=0.908691/2.3, alarm hl:mem_free=190.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.744629/1.9, alarm hl:np_load_long=1.008301/2.25, alarm hl:mem_free=190.000000M/200M, alarm hl:available=1/0 [00:21:03] [[w:en:User:Madman]] * Re: [Toolserver-l] Dementia (was: 403: User account expired toolserver.org/~soxred93) [00:23:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97677.000000 [00:25:26] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.002930/1.00, alarm hl:np_load_long=0.967773/1.50, alarm hl:mem_free=20161.000000M/300M, alarm hl:available=1/0 [00:26:26] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:31:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:49:58] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:51:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:51:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:53:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:54:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 455437 MB (8% inode=44%): [00:54:36] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.040527/1.8, alarm hl:np_load_avg=1.006348/2.3, alarm hl:mem_free=136.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.040527/1.9, alarm hl:np_load_long=0.918945/2.25, alarm hl:mem_free=136.000000M/200M, alarm hl:available=1/0 [00:55:05] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:26] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:27] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:35] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:36] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:55:48] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:59] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:06] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:57:04] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [00:57:05] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [00:57:17] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [00:57:18] SMTP on z-dat-s7-a is OK: SMTP OK - 1.566 sec. response time [00:57:19] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:57:19] MySQL slave on z-dat-s7-a is OK: Uptime: 2216044 Threads: 6 Questions: 491165853 Slow queries: 75347 Opens: 3527350 Flush tables: 1 Open tables: 6918 Queries per second avg: 221.640 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 277 [00:57:19] SMF on z-dat-s4-a is OK: OK - all services online [00:57:26] MySQL on z-dat-s7-a is OK: Uptime: 2216049 Threads: 5 Questions: 491167290 Slow queries: 75347 Opens: 3527350 Flush tables: 1 Open tables: 6918 Queries per second avg: 221.640 [00:57:27] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [00:57:27] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:57:27] MySQL slave on z-dat-s3-a is OK: Uptime: 1783631 Threads: 11 Questions: 1985656334 Slow queries: 119876 Opens: 13724347 Flush tables: 1 Open tables: 16384 Queries per second avg: 1113.266 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 253 [00:57:27] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:57:36] MySQL on z-dat-s3-a is OK: Uptime: 1783638 Threads: 10 Questions: 1985658787 Slow queries: 119876 Opens: 13724351 Flush tables: 1 Open tables: 16384 Queries per second avg: 1113.263 [00:57:36] SMF on z-dat-s6-a is OK: OK - all services online [00:57:57] SMTP on hyacinth is OK: SMTP OK - 0.007 sec. response time [00:59:36] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.065918/1.8, alarm hl:np_load_avg=1.054688/2.3, alarm hl:mem_free=243.000000M/300M, alarm hl:available=1/0 [01:02:36] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:19:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101055.000000 [01:23:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101289.000000 [01:38:46] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.650879/1.8, alarm hl:np_load_avg=0.711426/2.3, alarm hl:mem_free=270.000000M/300M, alarm hl:available=1/0 [01:42:46] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:49:46] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.901856/1.8, alarm hl:np_load_avg=0.834473/2.3, alarm hl:mem_free=187.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.901856/1.9, alarm hl:np_load_long=0.830566/2.25, alarm hl:mem_free=187.000000M/200M, alarm hl:available=1/0 [01:49:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:50:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:51:55] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:52:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:54:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:54:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 457155 MB (8% inode=44%): [01:55:37] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:38] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:38] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:39] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:46] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:46] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:56] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:57] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:05] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:06] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:06] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:06] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:07] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:07] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:07] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:07] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:07] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:08] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:08] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:15] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:15] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:15] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:15] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:27] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:27] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:27] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:27] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:41] [[Special:Log/newusers]] create 10 * Sathappan.spm * (New user account) [01:56:47] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [01:57:47] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [01:57:58] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [01:57:58] SMF on z-dat-s4-a is OK: OK - all services online [01:57:58] Load avg. on z-dat-s4-a is OK: OK - load average: 0.49, 1.12, 1.83 [01:57:58] / on z-dat-s3-a is OK: DISK OK - free space: / 11635 MB (38% inode=87%): [01:57:58] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 2793 MB (99% inode=99%): [01:57:58] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2787 MB (99% inode=99%): [01:57:58] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2794 MB (99% inode=99%): [01:57:59] / on z-dat-s6-a is OK: DISK OK - free space: / 11635 MB (38% inode=87%): [01:57:59] SMF on z-dat-s7-a is OK: OK - all services online [01:58:00] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 199465 MB (20% inode=99%): [01:58:00] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 115121 MB (28% inode=99%): [01:58:01] / on z-dat-s4-a is OK: DISK OK - free space: / 11635 MB (38% inode=87%): [01:58:01] SMF on z-dat-s6-a is OK: OK - all services online [01:58:02] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 113299 MB (28% inode=99%): [01:58:26] SMTP on z-dat-s6-a is OK: SMTP OK - 0.006 sec. response time [01:58:26] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:58:27] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:58:27] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 199515 MB (20% inode=99%): [01:58:27] Load avg. on z-dat-s6-a is OK: OK - load average: 1.27, 1.24, 1.84 [01:58:27] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:58:36] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:58:36] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:19:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104663.000000 [02:23:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104890.000000 [02:29:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.691894/1.8, alarm hl:np_load_avg=0.793945/2.3, alarm hl:mem_free=246.000000M/300M, alarm hl:available=1/0 [02:30:25] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1951 [02:30:25] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1950 [02:31:48] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:34:46] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.442383/1.10, alarm hl:np_load_long=1.004883/1.55, alarm hl:mem_free=21497.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.442383/1.00, alarm hl:np_load_long=1.004883/1.50, alarm hl:mem_free=21497.000000M/300M, alarm hl:available=1/0 [02:36:10] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:38:09] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.677246/1.8, alarm hl:np_load_avg=0.806641/2.3, alarm hl:mem_free=176.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.677246/1.9, alarm hl:np_load_long=0.903809/2.25, alarm hl:mem_free=176.000000M/200M, alarm hl:available=1/0 [02:42:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:48:10] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.352539/1.10, alarm hl:np_load_long=0.397949/1.55, alarm hl:mem_free=299.000000M/300M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.352539/1.00, alarm hl:np_load_long=0.397949/1.50, alarm hl:mem_free=299.000000M/300M, alarm hl:available=1/0 [02:50:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:50:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:50:09] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [02:52:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:52:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:54:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:55:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 454883 MB (8% inode=44%): [02:58:17] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3631 [02:58:17] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3630 [03:01:47] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:04:18] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201114 MB (20% inode=99%): [03:06:18] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:13:10] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.149414/1.10, alarm hl:np_load_long=0.919922/1.55, alarm hl:mem_free=21143.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.149414/1.00, alarm hl:np_load_long=0.919922/1.50, alarm hl:mem_free=21143.000000M/300M, alarm hl:available=1/0 [03:15:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:19:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108266.000000 [03:24:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108550.000000 [03:50:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:50:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:52:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:52:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:54:17] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:56:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 454418 MB (8% inode=44%): [03:58:18] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7231 [03:58:18] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7230 [04:02:17] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.034180/1.00, alarm hl:np_load_long=0.902344/1.50, alarm hl:mem_free=20975.000000M/300M, alarm hl:available=1/0 [04:03:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.139649/1.8, alarm hl:np_load_avg=1.032715/2.3, alarm hl:mem_free=246.000000M/300M, alarm hl:available=1/0 [04:03:18] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:05:08] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:13:30] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.035156/1.00, alarm hl:np_load_long=0.930664/1.50, alarm hl:mem_free=21178.000000M/300M, alarm hl:available=1/0 [04:18:07] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.607422/1.8, alarm hl:np_load_avg=0.781250/2.3, alarm hl:mem_free=211.000000M/300M, alarm hl:available=1/0 [04:20:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 111930.000000 [04:22:57] RAID on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:23:27] RAID on thyme is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:24:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112158.000000 [04:30:13] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6911&oldid=6850&rcid=9103 * 91.198.174.202 * (+0) (updated page) [04:36:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:50:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:52:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:52:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:54:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:56:27] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 453948 MB (8% inode=44%): [04:58:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10845 [04:58:33] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10845 [05:02:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.130371/1.8, alarm hl:np_load_avg=0.897461/2.3, alarm hl:mem_free=159.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.130371/1.9, alarm hl:np_load_long=0.764648/2.25, alarm hl:mem_free=159.000000M/200M, alarm hl:available=1/0 [05:04:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:07:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.816894/1.8, alarm hl:np_load_avg=0.859375/2.3, alarm hl:mem_free=276.000000M/300M, alarm hl:available=1/0 [05:21:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115567.000000 [05:24:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115765.000000 [05:50:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:50:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:52:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:53:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:54:45] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:56:11] [[Special:Log/newusers]] create 10 * Zt610152145 * (New user account) [05:57:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 451370 MB (8% inode=43%): [05:59:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14505 [05:59:33] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14505 [06:02:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.528809/1.8, alarm hl:np_load_avg=1.335449/2.3, alarm hl:mem_free=236.000000M/300M, alarm hl:available=1/0 [06:04:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:07:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.090820/1.8, alarm hl:np_load_avg=1.224609/2.3, alarm hl:mem_free=169.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.090820/1.9, alarm hl:np_load_long=1.123047/2.25, alarm hl:mem_free=169.000000M/200M, alarm hl:available=1/0 [06:13:12] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 15.781 second response time [06:21:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119168.000000 [06:22:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:24:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119368.000000 [06:34:46] [[Special:Log/newusers]] create 10 * Ykvach * (New user account) [06:51:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:51:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:52:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:53:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:54:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:58:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450887 MB (8% inode=43%): [06:59:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18109 [06:59:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18109 [07:02:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.576660/1.8, alarm hl:np_load_avg=1.375000/2.3, alarm hl:mem_free=225.000000M/300M, alarm hl:available=1/0 [07:03:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:10:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.325195/1.8, alarm hl:np_load_avg=1.458496/2.3, alarm hl:mem_free=237.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.325195/1.9, alarm hl:np_load_long=1.303223/2.25, alarm hl:mem_free=237.000000M/200M, alarm hl:available=1/0 [07:22:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122828.000000 [07:22:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:25:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123030.000000 [07:34:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.161133/1.8, alarm hl:np_load_avg=1.167480/2.3, alarm hl:mem_free=290.000000M/300M, alarm hl:available=1/0 [07:36:12] [[Main Page/lang]] ! 10https://wiki.toolserver.org/w/index.php?diff=6912&oldid=6181&rcid=9106 * Ykvach * (-456) () [07:49:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:51:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:51:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:52:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:53:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:54:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:58:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450330 MB (8% inode=43%): [07:59:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21718 [07:59:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21719 [08:12:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.132324/1.8, alarm hl:np_load_avg=1.222656/2.3, alarm hl:mem_free=283.000000M/300M, alarm hl:available=1/0 [08:13:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:22:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126429.000000 [08:22:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:25:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126639.000000 [08:34:54] MySQL on thyme is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [08:51:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:51:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:52:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:53:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:54:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:58:42] Hello all [08:59:05] Are there informations about s1 - replag? [08:59:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449853 MB (8% inode=43%): [08:59:42] Was there an announce i have overlooked [08:59:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25319 [09:00:24] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [09:09:52] MySQL on thyme is OK: Uptime: 603 Threads: 1 Questions: 2 Slow queries: 0 Opens: 18 Flush tables: 1 Open tables: 9 Queries per second avg: 0.3 [09:10:23] MySQL slave on thyme is OK: Uptime: 629 Threads: 1 Questions: 18 Slow queries: 0 Opens: 20 Flush tables: 1 Open tables: 11 Queries per second avg: 0.28 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [09:10:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.652832/1.8, alarm hl:np_load_avg=0.934570/2.3, alarm hl:mem_free=181.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.652832/1.9, alarm hl:np_load_long=0.846680/2.25, alarm hl:mem_free=181.000000M/200M, alarm hl:available=1/0 [09:12:42] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:32] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:33] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:03] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:03] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:25] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:25] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:25] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:14:42] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:14:54] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [09:15:12] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:15:24] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [09:15:25] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:25] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:33] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:33] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:42] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [09:15:42] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [09:15:42] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [09:15:42] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:42] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:43] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:55] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [09:15:55] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:55] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:55] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:55] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:55] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:55] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:56] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:15:56] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:16:02] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [09:16:03] MySQL slave on z-dat-s3-a is OK: Uptime: 1813545 Threads: 30 Questions: 2008223829 Slow queries: 120865 Opens: 13935520 Flush tables: 1 Open tables: 16384 Queries per second avg: 1107.347 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 302 [09:16:03] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:16:03] SMF on hyacinth is OK: OK - all services online [09:16:03] SMF on z-dat-s7-a is OK: OK - all services online [09:16:03] SMF on z-dat-s3-a is OK: OK - all services online [09:16:03] SMTP on z-dat-s6-a is OK: SMTP OK - 8.848 sec. response time [09:16:03] MySQL slave on z-dat-s6-a is OK: Uptime: 1722437 Threads: 17 Questions: 262507889 Slow queries: 116521 Opens: 2889803 Flush tables: 2 Open tables: 2760 Queries per second avg: 152.404 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 287 [09:16:04] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 199022 MB (20% inode=99%): [09:16:04] MySQL on z-dat-s7-a is OK: Uptime: 2245970 Threads: 9 Questions: 504117639 Slow queries: 75992 Opens: 3668698 Flush tables: 1 Open tables: 6914 Queries per second avg: 224.454 [09:16:05] MySQL slave on z-dat-s7-a is OK: Uptime: 2245971 Threads: 9 Questions: 504117814 Slow queries: 75992 Opens: 3668709 Flush tables: 1 Open tables: 6914 Queries per second avg: 224.454 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 270 [09:16:05] MySQL slave on z-dat-s4-a is OK: Uptime: 1722438 Threads: 7 Questions: 71836877 Slow queries: 24157 Opens: 45302 Flush tables: 1 Open tables: 512 Queries per second avg: 41.706 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 272 [09:16:25] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:16:25] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:16:25] / on z-dat-s3-a is OK: DISK OK - free space: / 11634 MB (38% inode=87%): [09:16:25] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:16:25] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 113056 MB (28% inode=99%): [09:16:25] Load avg. on z-dat-s4-a is OK: OK - load average: 1.13, 1.21, 1.72 [09:16:26] / on z-dat-s4-a is OK: DISK OK - free space: / 11634 MB (38% inode=87%): [09:16:26] SMTP on hyacinth is OK: SMTP OK - 0.002 sec. response time [09:16:26] Load avg. on z-dat-s3-a is OK: OK - load average: 1.16, 1.21, 1.72 [09:16:27] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 199019 MB (20% inode=99%): [09:16:27] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2860 MB (99% inode=99%): [09:16:28] SMF on z-dat-s4-a is OK: OK - all services online [09:16:32] MySQL on z-dat-s3-a is OK: Uptime: 1813576 Threads: 27 Questions: 2008232953 Slow queries: 120885 Opens: 13935537 Flush tables: 1 Open tables: 16384 Queries per second avg: 1107.333 [09:16:52] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:22:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130029.000000 [09:22:41] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:25:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130242.000000 [09:27:25] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:32] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:42] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:38:32] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:32] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:41] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:38:42] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:38:42] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:12] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:39:12] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3037 MB (99% inode=99%): [09:39:12] Load avg. on z-dat-s7-a is OK: OK - load average: 1.02, 1.20, 1.47 [09:39:12] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3038 MB (99% inode=99%): [09:39:23] MySQL on z-dat-s3-a is OK: Uptime: 1814942 Threads: 16 Questions: 2008740724 Slow queries: 121049 Opens: 13935997 Flush tables: 1 Open tables: 16384 Queries per second avg: 1106.779 [09:39:23] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:39:23] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:44:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.222656/1.10, alarm hl:np_load_long=0.755860/1.55, alarm hl:mem_free=21536.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.222656/1.00, alarm hl:np_load_long=0.755860/1.50, alarm hl:mem_free=21536.000000M/300M, alarm hl:available=1/0 [09:45:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:45:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.831055/1.8, alarm hl:np_load_avg=0.732422/2.3, alarm hl:mem_free=275.000000M/300M, alarm hl:available=1/0 [09:49:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:52:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:52:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:53:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:54:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:55:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:59:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449284 MB (8% inode=43%): [09:59:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28921 [10:02:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.876465/1.8, alarm hl:np_load_avg=0.903320/2.3, alarm hl:mem_free=197.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.876465/1.9, alarm hl:np_load_long=0.788574/2.25, alarm hl:mem_free=197.000000M/200M, alarm hl:available=1/0 [10:22:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133631.000000 [10:22:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:26:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133902.000000 [10:42:17] Is anyone interested in finding bad page titles on hi-wp that were created after namespace shortcuts were added? [10:44:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.455078/1.10, alarm hl:np_load_long=0.850586/1.55, alarm hl:mem_free=21411.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.455078/1.00, alarm hl:np_load_long=0.850586/1.50, alarm hl:mem_free=21411.000000M/300M, alarm hl:available=1/0 [10:46:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:53:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:53:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:53:53] anyone? [10:54:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:54:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:55:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:55:53] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:56:25] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:00:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448731 MB (8% inode=43%): [11:00:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32581 [11:12:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.966797/1.8, alarm hl:np_load_avg=0.810059/2.3, alarm hl:mem_free=175.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.966797/1.9, alarm hl:np_load_long=0.736816/2.25, alarm hl:mem_free=175.000000M/200M, alarm hl:available=1/0 [11:13:25] Sid-G: see the topic -- SQL query request service: http://bit.ly/9fqWG1 [11:14:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:14:05] looking [11:21:38] valhallasw:I'd rather wait for someone to get round to it here than file a JIRA ticket [11:22:00] valhallasw:I could reopen a bug though [11:22:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 137233.000000 [11:22:52] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:24:37] valhallasw:Its a small request btw [11:24:43] that's what they all say [11:25:11] and considering you haven't formulated your question clearly, it's going to take some time to find out what you actually want :p [11:25:53] heh, seriously see https://bugzilla.wikimedia.org/show_bug.cgi?id=35368 [11:25:53] valhallasw:U'll see how small it is [11:26:23] valhallasw:check for inaccessible pages [11:26:38] see the namespace names activated at http://hi.wikipedia.org/wiki/%E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A4%BF%E0%A4%AA%E0%A5%80%E0%A4%A1%E0%A4%BF%E0%A4%AF%E0%A4%BE:%E0%A4%A8%E0%A4%AF%E0%A5%80_%E0%A4%B8%E0%A5%81%E0%A4%B5%E0%A4%BF%E0%A4%A7%E0%A4%BE%E0%A4%8F%E0%A4%81 [11:27:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 137509.000000 [11:27:34] some pages in the mainspace contained the namespace names or shortcuts before the shortcuts were added [11:28:14] now, they just resolve to the namespace instead of the page in the namespace, making the pages inaccessible [11:28:14] valhallasw:That clear enough? [11:29:24] now write that down in a query request :p [11:29:50] noting that there already is such a query in namespaceDupes.php might also help for anyone who runs it for you [11:30:40] last but not least, *explicitly* listing the aliases instead of requiring someone to understand that unclear hiwiki page will also help [11:31:03] valhallasw:I'm not creating a JIRA account, even if I have to make do with bad page titles [11:35:52] ...whatever. [11:45:03] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.097656/1.00, alarm hl:np_load_long=0.811523/1.50, alarm hl:mem_free=21583.000000M/300M, alarm hl:available=1/0 [11:46:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:53:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:53:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:54:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:55:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:55:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:00:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448187 MB (8% inode=43%): [12:00:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36184 [12:13:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.643066/1.8, alarm hl:np_load_avg=0.698242/2.3, alarm hl:mem_free=203.000000M/300M, alarm hl:available=1/0 [12:14:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:23:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140893.000000 [12:27:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 141111.000000 [12:36:52] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:45:12] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.110351/1.10, alarm hl:np_load_long=0.827148/1.55, alarm hl:mem_free=20838.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.110351/1.00, alarm hl:np_load_long=0.827148/1.50, alarm hl:mem_free=20838.000000M/300M, alarm hl:available=1/0 [12:46:13] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:53:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:53:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:54:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:55:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:55:35] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:00:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447635 MB (8% inode=43%): [13:00:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39785 [13:21:42] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:23:35] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:23:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144501.000000 [13:27:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144720.000000 [13:36:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:43:13] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.425781/1.10, alarm hl:np_load_long=0.766602/1.55, alarm hl:mem_free=21043.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.425781/1.00, alarm hl:np_load_long=0.766602/1.50, alarm hl:mem_free=21043.000000M/300M, alarm hl:available=1/0 [13:46:13] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:53:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:53:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:54:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:55:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:55:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:00:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447123 MB (8% inode=43%): [14:01:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43403 [14:01:26] Hello all [14:04:32] @replag [14:04:32] DaBPunkt: s1-rr-a: 1d 16h 49m 28s [+1.00 s/s]; s1-user: 1d 16h 49m 28s [+1.00 s/s]; s2-user: 12s [-0.00 s/s]; s3-rr-a: 50s [+0.00 s/s]; s3-user: 50s [+0.00 s/s] [14:23:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 148108.000000 [14:23:43] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:28:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 148383.000000 [14:47:54] 3(commented) [ACCAPP-480] Account Request TU Vienna, Student member (PhD) <10https://jira.toolserver.org/browse/ACCAPP-480> (Kolossos ) [14:52:52] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 202756 MB (20% inode=99%): [14:53:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:53:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:55:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:55:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:55:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:00:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450743 MB (8% inode=43%): [15:02:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47066 [15:04:32] /sql on thyme is WARNING: DISK WARNING - free space: /sql 129991 MB (13% inode=99%): [15:12:56] [[Hlavní strana]] ! 10https://wiki.toolserver.org/w/index.php?diff=6913&oldid=5805&rcid=9107 * Mormegil * (+196) (+podporující WM) [15:17:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.492188/1.8, alarm hl:np_load_avg=0.484375/2.3, alarm hl:mem_free=281.000000M/300M, alarm hl:available=1/0 [15:18:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:23:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151719.000000 [15:24:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:28:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151988.000000 [15:32:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.620606/1.8, alarm hl:np_load_avg=0.593750/2.3, alarm hl:mem_free=229.000000M/300M, alarm hl:available=1/0 [15:44:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.378906/1.10, alarm hl:np_load_long=0.829102/1.55, alarm hl:mem_free=21331.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.378906/1.00, alarm hl:np_load_long=0.829102/1.50, alarm hl:mem_free=21331.000000M/300M, alarm hl:available=1/0 [15:47:31] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:49:56] 3(created) [MAGNUS-308] Catscan broken on plwiki; Magnus' tools; Bug <10https://jira.toolserver.org/browse/MAGNUS-308> (bulwersator) [15:53:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.466797/1.10, alarm hl:np_load_long=0.938477/1.55, alarm hl:mem_free=21337.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.466797/1.00, alarm hl:np_load_long=0.938477/1.50, alarm hl:mem_free=21337.000000M/300M, alarm hl:available=1/0 [15:53:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:53:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:55:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:55:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:55:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:00:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:00:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450165 MB (8% inode=43%): [16:02:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50669 [16:03:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.464844/1.8, alarm hl:np_load_avg=0.471680/2.3, alarm hl:mem_free=142.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.464844/1.9, alarm hl:np_load_long=0.459473/2.25, alarm hl:mem_free=142.000000M/200M, alarm hl:available=1/0 [16:07:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:24:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 155379.000000 [16:25:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:28:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 155590.000000 [16:31:52] 3(created) [UTRS-88] No field for block information; UTRS: Main Interface; New Feature <10https://jira.toolserver.org/browse/UTRS-88> (Chris Howie) [16:31:54] 3(updated) [UTRS-88] Add field for block information <10https://jira.toolserver.org/browse/UTRS-88> (Chris Howie) [16:36:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.167969/1.10, alarm hl:np_load_long=0.915039/1.55, alarm hl:mem_free=21223.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.167969/1.00, alarm hl:np_load_long=0.915039/1.50, alarm hl:mem_free=21223.000000M/300M, alarm hl:available=1/0 [16:39:40] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:41:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.541016/1.8, alarm hl:np_load_avg=0.488770/2.3, alarm hl:mem_free=235.000000M/300M, alarm hl:available=1/0 [16:45:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:49:53] 3(commented) [UTRS-79] Sent email content in an appeal's log discloses the appealing user's email address <10https://jira.toolserver.org/browse/UTRS-79> (Hersfold) [16:53:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:53:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:55:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:56:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:56:51] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:01:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449151 MB (8% inode=43%): [17:03:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54331 [17:03:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.509766/1.8, alarm hl:np_load_avg=0.471680/2.3, alarm hl:mem_free=212.000000M/300M, alarm hl:available=1/0 [17:05:53] 3(commented) [UTRS-88] Add field for block information <10https://jira.toolserver.org/browse/UTRS-88> (Hersfold) [17:07:52] 3(commented) [UTRS-88] Add field for block information <10https://jira.toolserver.org/browse/UTRS-88> (Chris Howie) [17:11:00] 3(commented) [UTRS-88] Add field for block information <10https://jira.toolserver.org/browse/UTRS-88> (Hersfold) [17:12:53] 3(updated) [UTRS-88] Display IP address in all cases when user does not have an account <10https://jira.toolserver.org/browse/UTRS-88> (Hersfold) [17:13:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125000/1.10, alarm hl:np_load_long=0.882812/1.55, alarm hl:mem_free=20639.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125000/1.00, alarm hl:np_load_long=0.882812/1.50, alarm hl:mem_free=20639.000000M/300M, alarm hl:available=1/0 [17:14:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:16:40] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:25:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:25:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159039.000000 [17:26:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.501465/1.8, alarm hl:np_load_avg=0.459473/2.3, alarm hl:mem_free=186.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.501465/1.9, alarm hl:np_load_long=0.467285/2.25, alarm hl:mem_free=186.000000M/200M, alarm hl:available=1/0 [17:29:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159251.000000 [17:31:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:43:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.419922/1.10, alarm hl:np_load_long=0.866211/1.55, alarm hl:mem_free=21039.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.419922/1.00, alarm hl:np_load_long=0.866211/1.50, alarm hl:mem_free=21039.000000M/300M, alarm hl:available=1/0 [17:45:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:47:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.153320/1.8, alarm hl:np_load_avg=0.720215/2.3, alarm hl:mem_free=137.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.153320/1.9, alarm hl:np_load_long=0.547363/2.25, alarm hl:mem_free=137.000000M/200M, alarm hl:available=1/0 [17:53:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.350586/1.10, alarm hl:np_load_long=0.980469/1.55, alarm hl:mem_free=21070.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.350586/1.00, alarm hl:np_load_long=0.980469/1.50, alarm hl:mem_free=21070.000000M/300M, alarm hl:available=1/0 [17:53:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:53:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:55:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:56:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:56:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:01:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448581 MB (8% inode=43%): [18:04:21] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57991 [18:16:54] 3(assigned) [UTRS-83] Comments cut off in appeals <10https://jira.toolserver.org/browse/UTRS-83> (Andrew Pearson) [18:16:57] 3(assigned) [UTRS-88] Display IP address in all cases when user does not have an account <10https://jira.toolserver.org/browse/UTRS-88> (Andrew Pearson) [18:18:55] 3(assigned) [UTRS-85] "void()" is not valid JavaScript <10https://jira.toolserver.org/browse/UTRS-85> (Andrew Pearson) [18:25:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:26:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162698.000000 [18:29:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162851.000000 [18:43:46] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.084961/1.00, alarm hl:np_load_long=0.817383/1.50, alarm hl:mem_free=20892.000000M/300M, alarm hl:available=1/0 [18:46:13] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:54:16] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.062500/1.00, alarm hl:np_load_long=0.855469/1.50, alarm hl:mem_free=21072.000000M/300M, alarm hl:available=1/0 [18:54:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:54:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:56:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:57:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:57:18] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:02:18] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448068 MB (8% inode=43%): [19:04:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61596 [19:18:14] nighty~ [19:25:41] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:27:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 166328.000000 [19:29:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 166459.000000 [19:41:52] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:54:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:54:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:57:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:57:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:57:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:02:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447500 MB (8% inode=43%): [20:05:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 65267 [20:06:22] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:15:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.843750/1.10, alarm hl:np_load_long=0.889648/1.55, alarm hl:mem_free=20904.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.843750/1.00, alarm hl:np_load_long=0.889648/1.50, alarm hl:mem_free=20904.000000M/300M, alarm hl:available=1/0 [20:20:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:26:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:27:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 169931.000000 [20:29:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170069.000000 [20:36:32] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.344726/1.10, alarm hl:np_load_long=0.385254/1.55, alarm hl:mem_free=238.000000M/300M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.344726/1.00, alarm hl:np_load_long=0.385254/1.50, alarm hl:mem_free=238.000000M/300M, alarm hl:available=1/0 [20:37:33] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.252 second response time [20:38:32] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.242 second response time [20:39:23] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [20:45:23] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.028320/1.00, alarm hl:np_load_long=0.871094/1.50, alarm hl:mem_free=21334.000000M/300M, alarm hl:available=1/0 [20:46:24] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:55:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:55:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:57:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:57:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:57:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:03:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 471833 MB (8% inode=45%): [21:05:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68868 [21:12:35] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.255859/1.10, alarm hl:np_load_long=0.869140/1.55, alarm hl:mem_free=20885.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.255859/1.00, alarm hl:np_load_long=0.869140/1.50, alarm hl:mem_free=20885.000000M/300M, alarm hl:available=1/0 [21:13:45] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:29:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 173681.000000 [21:55:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:55:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:56:02] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 11.570 second response time [21:57:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:57:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:57:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:58:13] @replag [21:58:14] Joan: s1-rr-a: 2d 43m 10s [+1.00 s/s]; s1-user: 2d 43m 10s [+1.00 s/s]; s2-user: 10s [-0.00 s/s] [22:03:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 471278 MB (8% inode=44%): [22:05:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 72479 [22:12:22] Load avg. on willow is WARNING: WARNING - load average: 20.94, 15.01, 10.75 [22:12:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.553223/1.8, alarm hl:np_load_avg=1.889160/2.3, alarm hl:mem_free=521.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.553223/1.9, alarm hl:np_load_long=1.354492/2.25, alarm hl:mem_free=521.000000M/200M, alarm hl:available=1/0 [22:14:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.123047/1.10, alarm hl:np_load_long=0.864258/1.55, alarm hl:mem_free=21247.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.123047/1.00, alarm hl:np_load_long=0.864258/1.50, alarm hl:mem_free=21247.000000M/300M, alarm hl:available=1/0 [22:15:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:21:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:26:23] Load avg. on willow is OK: OK - load average: 11.34, 14.13, 14.02 [22:28:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177205.000000 [22:29:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177281.000000 [22:32:23] Load avg. on willow is WARNING: WARNING - load average: 15.96, 14.72, 14.11 [22:32:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.864258/1.8, alarm hl:np_load_avg=1.818359/2.3, alarm hl:mem_free=995.000000M/300M, alarm hl:available=1/0 [22:33:46] @replag [22:33:46] DaBPunkt: s1-rr-a: 2d 1h 18m 42s [+1.00 s/s]; s1-user: 2d 1h 18m 42s [+1.00 s/s] [22:33:56] stil increasing… [22:35:34] DaBPunkt: will the replication automatically start again when the schema update is finished? [22:35:40] yes [22:35:53] the schema-update is part of the normal replication [22:36:04] we didn't start it by hand [22:36:20] but it gets pushed in one go after the full update on the master is done? [22:36:43] the thing I don't understand is why there is no replication at all at the moment [22:36:48] or at least: not for the revision table [22:37:42] oh, wait, or there is replication, but there are no new entries being placed while earlier revisions are not updated yet [22:37:50] I should think some more before I type :p [22:37:54] valhallasw: that's simple: Replication is just a long lists of commands. When 1 command is done, the next is executed, thean the next and so on. At the moment the command is "ALTER the table" [22:38:19] ah OK [22:38:54] 3(updated) [UTRS-78] Many links have interfering JavaScript <10https://jira.toolserver.org/browse/UTRS-78> (DeltaQuad) [22:40:56] 3(updated) [UTRS-86] The double-quote character is not HTML-encoded in JavaScript HTML attributes <10https://jira.toolserver.org/browse/UTRS-86> (DeltaQuad) [22:42:53] 3(reopened) [UTRS-79] Sent email content in an appeal's log discloses the appealing user's email address <10https://jira.toolserver.org/browse/UTRS-79> (DeltaQuad) [22:42:56] 3(commented) [UTRS-83] Comments cut off in appeals <10https://jira.toolserver.org/browse/UTRS-83> (DeltaQuad) [22:42:58] 3(assigned) [UTRS-83] Comments cut off in appeals <10https://jira.toolserver.org/browse/UTRS-83> (DeltaQuad) [22:47:56] 3(commented) [UTRS-88] Display IP address in all cases when user does not have an account <10https://jira.toolserver.org/browse/UTRS-88> (DeltaQuad) [22:56:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:56:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:56:22] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 106205 MB (10% inode=99%): [22:57:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:58:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:58:04] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:04:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 470729 MB (8% inode=44%): [23:06:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76092 [23:10:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.789551/1.8, alarm hl:np_load_avg=1.628906/2.3, alarm hl:mem_free=230.000000M/300M, alarm hl:available=1/0 [23:12:21] Load avg. on willow is WARNING: WARNING - load average: 18.10, 15.89, 13.48 [23:19:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:19:22] Load avg. on willow is OK: OK - load average: 13.19, 14.68, 13.89 [23:29:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180865.000000 [23:30:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180892.000000 [23:32:22] Load avg. on willow is WARNING: WARNING - load average: 16.00, 14.61, 13.84 [23:33:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.923340/1.8, alarm hl:np_load_avg=1.816895/2.3, alarm hl:mem_free=487.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.923340/1.9, alarm hl:np_load_long=1.729004/2.25, alarm hl:mem_free=487.000000M/200M, alarm hl:available=1/0 [23:34:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:37:55] nacht ts [23:51:52] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:53:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.540039/1.10, alarm hl:np_load_long=1.585938/1.55, alarm hl:mem_free=20320.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.540039/1.00, alarm hl:np_load_long=1.585938/1.50, alarm hl:mem_free=20320.000000M/300M, alarm hl:available=1/0 [23:53:32] Load avg. on ortelius is WARNING: WARNING - load average: 23.10, 13.32, 7.17 [23:55:33] Load avg. on ortelius is OK: OK - load average: 8.60, 11.77, 7.40 [23:56:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:56:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:56:51] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:57:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:58:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:58:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1