[00:00:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.72, 22.88, 20.82 [00:10:12] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:11:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:11:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:37:02] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.539062/1.10, alarm hl:np_load_long=0.473633/1.55, alarm hl:mem_free=439.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.539062/1.00, alarm hl:np_load_long=0.473633/1.50, alarm hl:mem_free=439.000000M/600M, alarm hl:available=1/0 [00:37:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:38:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5640 MB (18% inode=93%): [00:41:02] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [00:44:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:52:42] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 365473 MB (6% inode=38%): [00:55:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122855 MB (20% inode=99%): [00:56:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:59:43] Load avg. on willow is WARNING: WARNING - load average: 18.45, 18.60, 19.96 [01:00:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.53, 20.08, 20.39 [01:03:43] Load avg. on willow is WARNING: WARNING - load average: 16.61, 18.82, 19.86 [01:11:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:11:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:30:51] Load avg. on willow is WARNING: WARNING - load average: 23.75, 19.75, 19.33 [01:37:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:38:21] / on wolfsbane is WARNING: DISK WARNING - free space: / 5530 MB (18% inode=93%): [01:44:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:52:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 365384 MB (6% inode=38%): [01:56:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:56:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122788 MB (20% inode=99%): [02:02:31] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:11:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:11:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:17:22] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:30:51] Load avg. on willow is WARNING: WARNING - load average: 23.59, 20.28, 19.64 [02:37:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:38:55] [[Special:Log/newusers]] create 10 * PiRSquared17 * (New user account) [02:39:21] / on wolfsbane is WARNING: DISK WARNING - free space: / 5389 MB (17% inode=93%): [02:39:35] [[User:Josebryan13]] !M 10https://wiki.toolserver.org/w/index.php?diff=7148&oldid=7147&rcid=9484 * PiRSquared17 * (+11) (please delete this spam) [02:42:35] [[User:PiRSquared17]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7149&rcid=9485 * PiRSquared17 * (+37) (create page) [02:43:17] [[Special:Log/block]] block 10 * Dispenser * (blocked [[02User:Josebryan1310]] with an expiry time of 1 year (account creation disabled): Spamming links to external sites) [02:43:37] [[Special:Log/delete]] delete 10 * Dispenser * (deleted "[[02User:Josebryan1310]]": Link spamming) [02:44:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:52:32] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:52:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363601 MB (6% inode=38%): [02:56:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122725 MB (20% inode=99%): [02:57:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:11:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:12:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:16:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.90, 21.70, 20.00 [03:17:51] Load avg. on willow is WARNING: WARNING - load average: 20.18, 21.27, 19.96 [03:18:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.68, 21.76, 20.21 [03:38:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:39:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5247 MB (17% inode=93%): [03:45:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:53:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363443 MB (6% inode=38%): [03:57:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:57:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122558 MB (20% inode=99%): [03:59:03] Load avg. on willow is WARNING: WARNING - load average: 16.39, 18.57, 19.96 [04:01:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.40, 19.50, 20.12 [04:03:03] Load avg. on willow is WARNING: WARNING - load average: 17.45, 18.86, 19.82 [04:03:11] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=5.499023/1.10, alarm hl:np_load_long=1.369141/1.55, alarm hl:mem_free=14261.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=5.499023/1.00, alarm hl:np_load_long=1.369141/1.50, alarm hl:mem_free=14261.000000M/600M, alarm hl:available=1/0 [04:11:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:12:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:26:11] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.684082/1.00, alarm hl:np_load_long=0.552246/1.50, alarm hl:mem_free=534.000000M/600M, alarm hl:available=1/0 [04:28:11] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:33:12] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.651367/1.10, alarm hl:np_load_long=0.553711/1.55, alarm hl:mem_free=390.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.651367/1.00, alarm hl:np_load_long=0.553711/1.50, alarm hl:mem_free=390.000000M/600M, alarm hl:available=1/0 [04:39:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:40:21] / on wolfsbane is WARNING: DISK WARNING - free space: / 5100 MB (17% inode=93%): [04:46:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:48:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.02, 21.88, 20.24 [04:53:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363397 MB (6% inode=38%): [04:57:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:57:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122453 MB (20% inode=99%): [05:01:11] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:07:11] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.186524/1.10, alarm hl:np_load_long=1.267578/1.55, alarm hl:mem_free=14063.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.186524/1.00, alarm hl:np_load_long=1.267578/1.50, alarm hl:mem_free=14063.000000M/600M, alarm hl:available=1/0 [05:08:12] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:11:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:12:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:19:02] Load avg. on willow is WARNING: WARNING - load average: 17.09, 18.96, 19.87 [05:21:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.84, 20.25, 20.22 [05:23:02] Load avg. on willow is WARNING: WARNING - load average: 17.98, 19.52, 19.96 [05:39:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:40:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 4916 MB (16% inode=93%): [05:47:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:54:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363308 MB (6% inode=38%): [05:54:01] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.45, 22.46, 21.91 [05:57:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:58:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122222 MB (20% inode=99%): [06:01:29] / on wolfsbane is OK: DISK OK - free space: / 7028 MB (23% inode=93%): [06:08:19] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.859375/1.10, alarm hl:np_load_long=1.031250/1.55, alarm hl:mem_free=14918.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.859375/1.00, alarm hl:np_load_long=1.031250/1.50, alarm hl:mem_free=14918.000000M/600M, alarm hl:available=1/0 [06:12:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:13:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:13:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:15:02] Load avg. on willow is WARNING: WARNING - load average: 15.00, 17.73, 19.99 [06:31:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.32, 21.83, 20.29 [06:33:00] Load avg. on willow is WARNING: WARNING - load average: 16.68, 20.17, 19.88 [06:40:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:43:32] [[Special:Log/newusers]] create 10 * Coachingmainzsgfd * (New user account) [06:46:20] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.019531/1.00, alarm hl:np_load_long=0.925781/1.50, alarm hl:mem_free=15636.000000M/600M, alarm hl:available=1/0 [06:47:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:48:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:55:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363172 MB (6% inode=38%): [06:56:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.99, 22.80, 20.02 [06:57:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:58:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122070 MB (20% inode=99%): [06:58:03] Load avg. on willow is WARNING: WARNING - load average: 18.76, 21.29, 19.81 [06:59:01] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.43, 21.55, 20.01 [07:07:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.229492/1.10, alarm hl:np_load_long=1.000976/1.55, alarm hl:mem_free=15724.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.229492/1.00, alarm hl:np_load_long=1.000976/1.50, alarm hl:mem_free=15724.000000M/600M, alarm hl:available=1/0 [07:12:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:14:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:19:02] Load avg. on willow is WARNING: WARNING - load average: 25.69, 21.25, 19.92 [07:31:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.93, 21.72, 19.79 [07:32:00] Load avg. on willow is WARNING: WARNING - load average: 25.33, 21.88, 19.98 [07:33:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.85, 21.60, 20.00 [07:41:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:48:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:53:02] Load avg. on willow is WARNING: WARNING - load average: 14.39, 16.80, 17.79 [07:53:19] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.025391/1.00, alarm hl:np_load_long=0.860352/1.50, alarm hl:mem_free=16083.000000M/600M, alarm hl:available=1/0 [07:54:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:55:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363140 MB (6% inode=38%): [07:57:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:58:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121937 MB (20% inode=99%): [08:12:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:14:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:18:19] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.000976/1.00, alarm hl:np_load_long=0.866211/1.50, alarm hl:mem_free=16343.000000M/600M, alarm hl:available=1/0 [08:19:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:22:20] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.056641/1.00, alarm hl:np_load_long=0.944336/1.50, alarm hl:mem_free=16136.000000M/600M, alarm hl:available=1/0 [08:41:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:49:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:53:02] Load avg. on willow is WARNING: WARNING - load average: 14.89, 16.64, 16.26 [08:56:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363078 MB (6% inode=38%): [08:57:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:59:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121944 MB (20% inode=99%): [09:02:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:07:10] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:13:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:14:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:33:20] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.035156/1.00, alarm hl:np_load_long=0.793945/1.50, alarm hl:mem_free=16326.000000M/600M, alarm hl:available=1/0 [09:34:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:41:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:49:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:54:01] Load avg. on willow is WARNING: WARNING - load average: 14.43, 16.43, 16.45 [09:57:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 363029 MB (6% inode=38%): [09:58:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:59:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121788 MB (20% inode=99%): [10:13:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:14:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:15:32] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.409180/1.10, alarm hl:np_load_long=0.419434/1.55, alarm hl:mem_free=332.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.409180/1.00, alarm hl:np_load_long=0.419434/1.50, alarm hl:mem_free=332.000000M/600M, alarm hl:available=1/0 [10:16:31] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [10:41:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:41:54] 3(commented) [DRTRIGON-122] Additional options to config sum_disc.py dealing with thread header <10https://jira.toolserver.org/browse/DRTRIGON-122> (Xqt ) [10:49:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:50:03] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:51:41] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:12] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:13] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:13] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:13] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:13] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:13] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:20] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:33] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:33] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:40] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:41] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:41] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:41] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:41] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:50] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [10:53:02] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:53:03] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:53:03] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:53:12] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:21] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:53:22] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:53:22] SMF on z-dat-s4-a is OK: OK - all services online [10:53:22] SMF on hyacinth is OK: OK - all services online [10:53:34] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:53:34] SMF on z-dat-s3-a is OK: OK - all services online [10:53:34] SMF on z-dat-s6-a is OK: OK - all services online [10:53:51] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:53:51] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:53:51] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [10:53:51] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:53:51] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:54:03] Load avg. on willow is WARNING: WARNING - load average: 13.69, 16.05, 16.31 [10:56:52] / on z-dat-s7-a is OK: DISK OK - free space: / 8388 MB (28% inode=85%): [10:56:52] SMF on z-dat-s7-a is OK: OK - all services online [10:57:03] MySQL on z-dat-s3-a is OK: Uptime: 225373 Threads: 39 Questions: 238188465 Slow queries: 19554 Opens: 2538909 Flush tables: 1 Open tables: 16384 Queries per second avg: 1056.863 [10:57:03] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2129 [10:57:03] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:57:03] SMTP on hyacinth is OK: SMTP OK - 0.019 sec. response time [10:57:04] NTP on hyacinth is OK: NTP OK: Offset -0.006523 secs [10:57:04] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2450 MB (99% inode=99%): [10:57:04] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1883 [10:57:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 362377 MB (6% inode=38%): [10:57:04] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [10:57:05] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:57:05] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:57:06] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:57:20] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [10:57:32] MySQL on z-dat-s4-a is OK: Uptime: 225403 Threads: 21 Questions: 27975854 Slow queries: 4261 Opens: 4550 Flush tables: 1 Open tables: 537 Queries per second avg: 124.114 [10:57:32] MySQL on z-dat-s7-a is OK: Uptime: 225402 Threads: 6 Questions: 80058133 Slow queries: 5725 Opens: 682145 Flush tables: 1 Open tables: 5939 Queries per second avg: 355.179 [10:57:32] MySQL slave on z-dat-s4-a is OK: Uptime: 225403 Threads: 21 Questions: 27975855 Slow queries: 4261 Opens: 4550 Flush tables: 1 Open tables: 537 Queries per second avg: 124.114 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 402 [10:57:32] MySQL on z-dat-s6-a is OK: Uptime: 225402 Threads: 12 Questions: 36377166 Slow queries: 12639 Opens: 144217 Flush tables: 2 Open tables: 1722 Queries per second avg: 161.387 [10:57:32] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2074 [10:57:33] SMTP on z-dat-s3-a is OK: SMTP OK - 0.041 sec. response time [10:57:33] SMTP on z-dat-s7-a is OK: SMTP OK - 0.084 sec. response time [10:57:34] SMTP on z-dat-s6-a is OK: SMTP OK - 0.039 sec. response time [10:57:40] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:58:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:59:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121636 MB (20% inode=99%): [11:13:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:14:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:28:02] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [11:42:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:42:33] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.404297/1.10, alarm hl:np_load_long=0.807617/1.55, alarm hl:mem_free=16372.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.404297/1.00, alarm hl:np_load_long=0.807617/1.50, alarm hl:mem_free=16372.000000M/600M, alarm hl:available=1/0 [11:45:33] MySQL slave on z-dat-s6-a is OK: Uptime: 228284 Threads: 6 Questions: 36556507 Slow queries: 13135 Opens: 144244 Flush tables: 2 Open tables: 1723 Queries per second avg: 160.136 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1738 [11:47:02] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3510 [11:50:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:50:33] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:55:03] Load avg. on willow is WARNING: WARNING - load average: 15.42, 16.91, 17.34 [11:57:33] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3066 [11:58:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 362317 MB (6% inode=38%): [11:58:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:00:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121460 MB (20% inode=99%): [12:06:32] MySQL slave on z-dat-s7-a is OK: Uptime: 229544 Threads: 9 Questions: 80357234 Slow queries: 6649 Opens: 682199 Flush tables: 1 Open tables: 5963 Queries per second avg: 350.73 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1691 [12:10:03] MySQL slave on z-dat-s3-a is OK: Uptime: 229755 Threads: 30 Questions: 242693038 Slow queries: 21555 Opens: 2598344 Flush tables: 1 Open tables: 16384 Queries per second avg: 1056.312 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1789 [12:13:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:15:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:42:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:48:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1885.000000 [12:50:03] /sql on ptolemy is OK: DISK OK - free space: /sql 129420 MB (21% inode=99%): [12:50:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:55:03] Load avg. on willow is WARNING: WARNING - load average: 13.20, 14.65, 15.54 [12:58:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 362269 MB (6% inode=38%): [12:58:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:07:04] Platonides, around? [13:08:19] I'd also like to have a list of pages for Wikipedia:Featured picture candidates/Grosser Wollschweber Bombylius major detail.jpg pattern where files are following the "Wikipedia:Featured picture candidates/{{PAGENAME}} pattern [13:10:03] Load avg. on willow is OK: OK - load average: 11.59, 13.89, 14.93 [13:13:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:15:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:36:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1775.000000 [13:42:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:46:57] 3(resolved) [OSM-13] Add map layers from the Norwegian Mapping Authority <10https://jira.toolserver.org/browse/OSM-13> (Kolossos ) [13:47:31] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:51:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:52:11] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:58:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:59:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 362181 MB (6% inode=38%): [14:09:23] ToAruShiroiNeko: On which wiki? [14:13:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:16:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:21:50] ToAruShiroiNeko: http://enwp.org/Special:PrefixIndex/Wikipedia:Featured_picture_candidates/Image: is what you're after right? [14:42:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:51:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:52:32] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:58:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:59:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 362099 MB (6% inode=37%): [15:04:57] 3(updated) [ACCAPP-488] Solving links to disambiguation pages in Wikipedia via "Personalized" Crowdsourcing <10https://jira.toolserver.org/browse/ACCAPP-488> (Amr Ebaid) [15:13:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:16:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:20:08] Dispenser on en.wiki [15:20:31] I am looking for "http://enwp.org/Special:PrefixIndex/Wikipedia:Featured_picture_candidates/" where the rest is identical to the filename [15:20:54] Wikipedia:Featured_picture_candidates/Foo.jpg" .gif .svg. et [15:20:56] So where it excluded File: or Image:? [15:21:05] yes without those [15:21:13] I already have lists for File: and Image: [15:25:33] ToAruShiroiNeko: http://toolserver.org/~nikola/grep.php?pattern=Featured_picture_candidates%2F%28[^%3A]*[.][a-z]*%29%24&lang=en&wiki=wikipedia&ns=4 [15:32:13] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:42:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:46:09] Are there any known issue with SGE or cron on willow? [15:46:31] it appears regular stuff from ~cvn hasn't ran in hours (usually every few minutes) [15:51:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:52:12] Load avg. on willow is WARNING: WARNING - load average: 14.75, 15.05, 14.64 [15:53:13] Load avg. on willow is OK: OK - load average: 14.01, 14.78, 14.57 [15:53:15] Dispenser NIIICE! [15:53:22] is that your tool? [15:54:00] could I ask you to have a counter and also the option to output as a text file? [15:54:09] Nope, but its my next favorite tool to CatScan 2 [15:58:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:59:05] ah [15:59:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 362005 MB (6% inode=37%): [15:59:13] catscan? [15:59:29] http://toolserver.org/~magnus/catscan_rewrite.php [16:06:12] Load avg. on willow is WARNING: WARNING - load average: 17.02, 15.69, 14.92 [16:09:30] I do not see pictures of cats [16:09:35] I am soo disapoint :p [16:10:30] Dispenser can catscan determine if a template is used multiple times? [16:11:11] No, the templatelinks table only describes which templates are used [16:13:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:16:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:16:41] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.436524/1.00, alarm hl:np_load_long=0.438476/1.50, alarm hl:mem_free=599.000000M/600M, alarm hl:available=1/0 [16:18:41] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [16:32:41] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.386230/1.00, alarm hl:np_load_long=0.369629/1.50, alarm hl:mem_free=588.000000M/600M, alarm hl:available=1/0 [16:37:21] Load avg. on willow is OK: OK - load average: 14.55, 14.62, 14.98 [16:41:22] Load avg. on willow is WARNING: WARNING - load average: 15.94, 15.33, 15.18 [16:42:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:43:22] Load avg. on willow is OK: OK - load average: 14.00, 14.67, 14.94 [16:51:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:52:31] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:57:32] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:58:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:59:21] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 361957 MB (6% inode=37%): [17:13:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:16:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:32:42] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.311035/1.00, alarm hl:np_load_long=0.342774/1.50, alarm hl:mem_free=573.000000M/600M, alarm hl:available=1/0 [17:33:42] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [17:41:42] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.398438/1.10, alarm hl:np_load_long=0.343750/1.55, alarm hl:mem_free=126.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.398438/1.00, alarm hl:np_load_long=0.343750/1.50, alarm hl:mem_free=126.000000M/600M, alarm hl:available=1/0 [17:42:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:44:49] [[Monthly report/2012-04]] ! 10https://wiki.toolserver.org/w/index.php?diff=7150&oldid=7146&rcid=9489 * 82.74.194.195 * (+14) (fix german-format date, https, space) [17:47:33] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:51:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:59:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:59:21] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 361810 MB (6% inode=37%): [18:13:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:16:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:42:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:52:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:54:09] hello all [18:59:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:59:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 361733 MB (6% inode=37%): [19:14:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:16:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:30:58] 3(commented) [ACCAPP-460] I want run a robot for fawiki, and im want mostly to faster doing job. <10https://jira.toolserver.org/browse/ACCAPP-460> (Mahdi haji) [19:32:21] Load avg. on willow is WARNING: WARNING - load average: 16.27, 13.58, 11.68 [19:33:21] Load avg. on willow is OK: OK - load average: 12.12, 12.78, 11.51 [19:33:52] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.576660/1.10, alarm hl:np_load_long=0.466309/1.55, alarm hl:mem_free=493.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.576660/1.00, alarm hl:np_load_long=0.466309/1.50, alarm hl:mem_free=493.000000M/600M, alarm hl:available=1/0 [19:42:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:44:01] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.252930/1.10, alarm hl:np_load_long=0.855469/1.55, alarm hl:mem_free=16630.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.252930/1.00, alarm hl:np_load_long=0.855469/1.50, alarm hl:mem_free=16630.000000M/600M, alarm hl:available=1/0 [19:45:02] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [19:45:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:49:02] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.500488/1.10, alarm hl:np_load_long=0.493164/1.55, alarm hl:mem_free=287.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.500488/1.00, alarm hl:np_load_long=0.493164/1.50, alarm hl:mem_free=287.000000M/600M, alarm hl:available=1/0 [19:52:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:59:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:59:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 361700 MB (6% inode=37%): [20:02:23] Load avg. on willow is WARNING: WARNING - load average: 15.33, 14.77, 12.75 [20:03:21] Load avg. on willow is OK: OK - load average: 12.21, 13.85, 12.55 [20:14:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:17:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:42:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:52:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:52:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:59:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:59:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 361633 MB (6% inode=37%): [21:03:29] [[Special:Log/newusers]] create 10 * Garybucnannon12 * (New user account) [21:14:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:16:15] Merlijn van Deen * Re: [Toolserver-l] Interwiki bot MMP planning [21:17:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:32:17] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:33:10] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=8.422852/1.10, alarm hl:np_load_long=1.480469/1.55, alarm hl:mem_free=17184.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=8.422852/1.00, alarm hl:np_load_long=1.480469/1.50, alarm hl:mem_free=17184.000000M/600M, alarm hl:available=1/0 [21:34:35] Load avg. on ortelius is WARNING: WARNING - load average: 28.66, 18.95, 9.16 [21:36:34] Load avg. on ortelius is OK: OK - load average: 6.76, 13.88, 8.46 [21:43:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:44:12] -join # [21:44:39] ooops, sorry