[00:00:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.613770/1.95, alarm hl:np_load_avg=1.438965/2.0, alarm hl:mem_free=143.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.613770/2.3, alarm hl:np_load_long=1.468750/2.5, alarm hl:cpu=90.800000/98, alarm hl:mem_free=143.000000M/150M, alarm hl:available=1/0 [00:02:41] Load avg. on willow is WARNING: WARNING - load average: 17.24, 15.20, 13.27 [00:10:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:23:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:23:41] Load avg. on willow is OK: OK - load average: 13.78, 14.79, 14.89 [00:27:30] [[User talk:Gifti]] ! 10https://wiki.toolserver.org/w/index.php?diff=7085&oldid=6963&rcid=9318 * Brandon Sky * (+702) (/* Request for undeletion */ ) [00:27:41] Load avg. on willow is WARNING: WARNING - load average: 16.63, 15.96, 15.36 [00:27:58] [[User talk:Gifti]] ! 10https://wiki.toolserver.org/w/index.php?diff=7086&oldid=7085&rcid=9319 * Brandon Sky * (+613) () [00:34:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:46:22] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [00:47:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:53:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 424045 MB (8% inode=41%): [01:00:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.189453/1.95, alarm hl:np_load_avg=2.178711/2.0, alarm hl:mem_free=435.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.189453/2.3, alarm hl:np_load_long=2.165039/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=435.000000M/150M, alarm hl:available=1/0 [01:00:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.21, 21.73, 18.84 [01:01:51] Load avg. on willow is WARNING: WARNING - load average: 23.33, 21.08, 18.80 [01:10:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:11:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.74, 24.03, 20.69 [01:22:01] Load avg. on willow is WARNING: WARNING - load average: 15.49, 18.72, 19.75 [01:23:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:34:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:46:25] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [01:47:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:53:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 422820 MB (7% inode=41%): [02:00:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.622559/1.95, alarm hl:np_load_avg=2.248047/2.0, alarm hl:mem_free=120.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.622559/2.3, alarm hl:np_load_long=2.306152/2.5, alarm hl:cpu=99.600000/98, alarm hl:mem_free=120.000000M/150M, alarm hl:available=1/0 [02:10:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:22:05] Load avg. on willow is WARNING: WARNING - load average: 17.40, 18.98, 18.45 [02:23:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:25:09] Joan: I want too block him too, maybe for signature abuse? [02:35:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:46:26] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [02:47:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:52:57] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:53:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 423268 MB (7% inode=41%): [02:55:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.363770/1.95, alarm hl:np_load_avg=2.030273/2.0, alarm hl:mem_free=520.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.363770/2.3, alarm hl:np_load_long=2.088867/2.5, alarm hl:cpu=99.900000/98, alarm hl:mem_free=520.000000M/150M, alarm hl:available=1/0 [02:58:55] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:02:11] Dispenser: I'm hoping he just gets bored quickly. [03:02:14] We need River. [03:02:18] She wouldn't hesitate to block. [03:03:53] [[User talk:Dispenser]] 10https://wiki.toolserver.org/w/index.php?diff=7087&oldid=7054&rcid=9320 * Dispenser * (+479) (/* Don't undo my edits */ I also am to be an admin here) [03:04:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.256836/1.10, alarm hl:np_load_long=0.794922/1.55, alarm hl:mem_free=17298.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.256836/1.00, alarm hl:np_load_long=0.794922/1.50, alarm hl:mem_free=17298.000000M/350M, alarm hl:available=1/0 [03:05:26] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:10:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:13:26] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.082031/1.00, alarm hl:np_load_long=0.841797/1.50, alarm hl:mem_free=17490.000000M/350M, alarm hl:available=1/0 [03:16:07] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.99, 22.41, 20.43 [03:20:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.659668/1.95, alarm hl:np_load_avg=2.652344/2.0, alarm hl:mem_free=657.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.659668/2.3, alarm hl:np_load_long=2.545898/2.5, alarm hl:cpu=99.900000/98, alarm hl:mem_free=657.000000M/150M, alarm hl:available=1/0 [03:22:07] Load avg. on willow is WARNING: WARNING - load average: 16.65, 19.90, 19.97 [03:23:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:35:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:46:37] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [03:47:36] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:49:35] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.151367/1.10, alarm hl:np_load_long=1.368164/1.55, alarm hl:mem_free=17872.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.151367/1.00, alarm hl:np_load_long=1.368164/1.50, alarm hl:mem_free=17872.000000M/350M, alarm hl:available=1/0 [03:53:37] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:54:06] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 423212 MB (7% inode=41%): [04:01:15] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.29, 22.06, 19.52 [04:02:15] Load avg. on willow is WARNING: WARNING - load average: 24.66, 21.96, 19.65 [04:03:37] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.032226/1.00, alarm hl:np_load_long=1.164062/1.50, alarm hl:mem_free=17792.000000M/350M, alarm hl:available=1/0 [04:10:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:21:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.250977/1.95, alarm hl:np_load_avg=2.342773/2.0, alarm hl:mem_free=330.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.250977/2.3, alarm hl:np_load_long=2.380371/2.5, alarm hl:cpu=98.400000/98, alarm hl:mem_free=330.000000M/150M, alarm hl:available=1/0 [04:23:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:35:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:46:46] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [04:47:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:53:46] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.040039/1.00, alarm hl:np_load_long=0.825195/1.50, alarm hl:mem_free=18377.000000M/350M, alarm hl:available=1/0 [04:54:46] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:55:06] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 423126 MB (7% inode=41%): [04:56:56] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:59:56] Sun Grid Engine execd on willow is WARNING: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.891113/2.3, alarm hl:np_load_long=2.040527/2.5, alarm hl:cpu=99.800000/98, alarm hl:mem_free=539.000000M/150M, alarm hl:available=1/0 [05:02:16] Load avg. on willow is WARNING: WARNING - load average: 17.79, 17.78, 17.21 [05:10:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:14:46] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.614258/1.10, alarm hl:np_load_long=0.984375/1.55, alarm hl:mem_free=18041.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.614258/1.00, alarm hl:np_load_long=0.984375/1.50, alarm hl:mem_free=18041.000000M/350M, alarm hl:available=1/0 [05:15:15] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:16:46] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:47] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:18:25] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [05:19:15] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.25, 23.27, 20.27 [05:23:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:24:55] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:31:20] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7088&oldid=6835&rcid=9321 * Brandon Sky * (+748) (/* Https red slash */ new section) [05:35:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:41:15] Load avg. on willow is WARNING: WARNING - load average: 18.94, 19.25, 19.93 [05:47:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:47:45] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [05:55:05] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 423039 MB (7% inode=41%): [06:00:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.643555/1.95, alarm hl:np_load_avg=2.517578/2.0, alarm hl:mem_free=470.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.643555/2.3, alarm hl:np_load_long=2.349609/2.5, alarm hl:cpu=98.000000/98, alarm hl:mem_free=470.000000M/150M, alarm hl:available=1/0 [06:10:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:23:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:25:04] @replag [06:25:05] matthewrbowker: s6-rr-a: 33s [-0.00 s/s]; s6-user: 33s [-0.00 s/s] [06:29:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:31:15] Load avg. on willow is CRITICAL: CRITICAL - load average: 39.12, 23.53, 20.08 [06:33:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.706543/1.95, alarm hl:np_load_avg=2.769531/2.0, alarm hl:mem_free=203.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.706543/2.3, alarm hl:np_load_long=2.491699/2.5, alarm hl:cpu=99.300000/98, alarm hl:mem_free=203.000000M/150M, alarm hl:available=1/0 [06:33:15] Load avg. on willow is WARNING: WARNING - load average: 19.31, 21.54, 19.79 [06:34:53] 3(commented) [ACC-247] Clicking on "Account Requests" generates a database query error <10https://jira.toolserver.org/browse/ACC-247> (Matthew Bowker) [06:34:53] 3(created) [ACC-247] Clicking on "Account Requests" generates a database query error; ACC: Internal interface; Blocker Bug <10https://jira.toolserver.org/browse/ACC-247> (Matthew Bowker) [06:35:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:37:54] 3(updated) [ACC-247] Clicking on "Account Requests" generates a database query error <10https://jira.toolserver.org/browse/ACC-247> (Matthew Bowker) [06:45:17] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.07, 20.91, 20.03 [06:47:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:47:46] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [06:55:16] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:16] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 422933 MB (7% inode=41%): [07:10:05] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:10:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:23:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:33:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.148926/1.95, alarm hl:np_load_avg=2.237793/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.148926/2.3, alarm hl:np_load_long=2.504883/2.5, alarm hl:cpu=99.000000/98, alarm hl:mem_free=125.000000M/150M, alarm hl:available=1/0 [07:33:25] Load avg. on willow is WARNING: WARNING - load average: 16.84, 17.79, 19.96 [07:35:26] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.38, 18.79, 20.03 [07:35:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:37:26] Load avg. on willow is WARNING: WARNING - load average: 18.37, 18.79, 19.90 [07:47:46] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [07:47:46] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:49:25] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.35, 21.12, 20.07 [07:55:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 422879 MB (7% inode=41%): [08:10:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:23:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:33:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.367676/1.95, alarm hl:np_load_avg=2.498535/2.0, alarm hl:mem_free=162.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.367676/2.3, alarm hl:np_load_long=2.591309/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=162.000000M/150M, alarm hl:available=1/0 [08:35:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:36:38] Load avg. on willow is WARNING: WARNING - load average: 17.43, 18.28, 19.86 [08:47:46] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [08:47:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:56:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 422720 MB (7% inode=41%): [09:04:48] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.286133/1.10, alarm hl:np_load_long=0.696289/1.55, alarm hl:mem_free=15953.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.286133/1.00, alarm hl:np_load_long=0.696289/1.50, alarm hl:mem_free=15953.000000M/350M, alarm hl:available=1/0 [09:10:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:12:45] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:15:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.19, 22.64, 20.28 [09:23:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:29:34] Load avg. on willow is WARNING: WARNING - load average: 16.03, 18.83, 19.77 [09:30:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.46, 20.12, 20.15 [09:34:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.355469/1.95, alarm hl:np_load_avg=2.484863/2.0, alarm hl:mem_free=333.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.355469/2.3, alarm hl:np_load_long=2.509766/2.5, alarm hl:cpu=96.100000/98, alarm hl:mem_free=333.000000M/150M, alarm hl:available=1/0 [09:35:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:36:36] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41273 MB (10% inode=99%): [09:39:36] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 50385 MB (12% inode=99%): [09:46:36] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40558 MB (9% inode=99%): [09:47:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:47:47] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [09:50:36] Load avg. on willow is WARNING: WARNING - load average: 20.46, 18.38, 18.27 [09:57:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 420670 MB (7% inode=41%): [10:10:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.31, 21.19, 18.91 [10:10:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:11:36] Load avg. on willow is WARNING: WARNING - load average: 23.04, 20.66, 18.88 [10:23:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:28:25] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:28:25] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:28:37] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:28:45] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:15] SMTP on z-dat-s6-a is OK: SMTP OK - 0.004 sec. response time [10:29:36] SMTP on z-dat-s4-a is OK: SMTP OK - 3.185 sec. response time [10:29:37] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:29:57] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:29:57] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:30:05] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:30:05] MySQL slave on z-dat-s3-a is OK: Uptime: 4409987 Threads: 19 Questions: 5022986800 Slow queries: 245025 Opens: 37824443 Flush tables: 1 Open tables: 16384 Queries per second avg: 1139.2 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95 [10:30:05] MySQL on z-dat-s3-a is OK: Uptime: 4409987 Threads: 19 Questions: 5022986802 Slow queries: 245025 Opens: 37824443 Flush tables: 1 Open tables: 16384 Queries per second avg: 1139.2 [10:34:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.765625/1.95, alarm hl:np_load_avg=2.129395/2.0, alarm hl:mem_free=99.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.765625/2.3, alarm hl:np_load_long=2.169922/2.5, alarm hl:cpu=89.700000/98, alarm hl:mem_free=99.000000M/150M, alarm hl:available=1/0 [10:36:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:42:37] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:47:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:47:57] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [10:57:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 422069 MB (7% inode=41%): [11:01:52] @replag [11:01:53] stwalkerster: s3-rr-a: 31s [-0.00 s/s]; s3-user: 31s [-0.00 s/s]; s7-rr-a: 14s [+0.00 s/s]; s7-user: 14s [+0.00 s/s] [11:02:25] I was beginning to think it wouldn't reply.... [11:03:06] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.474609/1.10, alarm hl:np_load_long=0.765625/1.55, alarm hl:mem_free=17954.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.474609/1.00, alarm hl:np_load_long=0.765625/1.50, alarm hl:mem_free=17954.000000M/350M, alarm hl:available=1/0 [11:04:55] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:07:56] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.103516/1.10, alarm hl:np_load_long=0.877930/1.55, alarm hl:mem_free=18108.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.103516/1.00, alarm hl:np_load_long=0.877930/1.50, alarm hl:mem_free=18108.000000M/350M, alarm hl:available=1/0 [11:11:00] 3(assigned) [ACC-247] Clicking on "Account Requests" generates a database query error <10https://jira.toolserver.org/browse/ACC-247> (Simon Walker) [11:11:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:11:45] Load avg. on willow is WARNING: WARNING - load average: 16.68, 17.92, 18.08 [11:24:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:30:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.82, 21.13, 18.71 [11:31:45] Load avg. on willow is WARNING: WARNING - load average: 26.05, 21.62, 19.05 [11:35:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.981934/1.95, alarm hl:np_load_avg=2.314941/2.0, alarm hl:mem_free=127.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.981934/2.3, alarm hl:np_load_long=2.285645/2.5, alarm hl:cpu=90.600000/98, alarm hl:mem_free=127.000000M/150M, alarm hl:available=1/0 [11:37:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:45:56] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.147461/1.10, alarm hl:np_load_long=0.974610/1.55, alarm hl:mem_free=17694.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.147461/1.00, alarm hl:np_load_long=0.974610/1.50, alarm hl:mem_free=17694.000000M/350M, alarm hl:available=1/0 [11:47:57] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:47:57] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [11:48:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:52:04] [[Special:Log/newusers]] create 10 * Karamat satti * (New user account) [11:57:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 421938 MB (7% inode=41%): [11:57:56] 3(resolved) [ACC-247] Clicking on "Account Requests" generates a database query error <10https://jira.toolserver.org/browse/ACC-247> (Simon Walker) [11:57:58] 3(resolved) [ACC-238] Ticket 68253 breaks our system! <10https://jira.toolserver.org/browse/ACC-238> (Simon Walker) [11:57:59] 3(closed) [ACC-247] Clicking on "Account Requests" generates a database query error <10https://jira.toolserver.org/browse/ACC-247> (Simon Walker) [12:00:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.88, 20.59, 18.48 [12:01:45] Load avg. on willow is WARNING: WARNING - load average: 25.77, 21.07, 18.79 [12:01:53] 3(updated) [ACC-228] ACC eating up IPs <10https://jira.toolserver.org/browse/ACC-228> (Simon Walker) [12:03:54] 3(resolved) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Marlen Caemmerer) [12:03:55] 3(resolved) [ACC-180] Add Betacommand's SIL tool to IP check list. <10https://jira.toolserver.org/browse/ACC-180> (Simon Walker) [12:03:57] 3(updated) [MNT-1202] Add more space to daphne <10https://jira.toolserver.org/browse/MNT-1202> (Marlen Caemmerer) [12:08:09] [[Special:Log/newusers]] create 10 * Kristeenlemarr58 * (New user account) [12:11:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:12:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.53, 23.32, 20.44 [12:20:52] [[User:Kristeenlemarr58]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7089&rcid=9324 * Kristeenlemarr58 * (+144) (Created page with "Mac Ragone m'a dit de bonne choses sur ce site web dont la thématique est film x My Site: [http://www.bigs.me/655daa film porno en streaming]") [12:24:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:24:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:32:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.082031/1.95, alarm hl:np_load_avg=1.972168/2.0, alarm hl:mem_free=616.000000M/350M, alarm hl:available=1/0 [12:32:45] Load avg. on willow is WARNING: WARNING - load average: 15.46, 15.58, 16.66 [12:33:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:37:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:44:05] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.10, alarm hl:np_load_long=0.730469/1.55, alarm hl:mem_free=16130.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.00, alarm hl:np_load_long=0.730469/1.50, alarm hl:mem_free=16130.000000M/350M, alarm hl:available=1/0 [12:45:05] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:48:05] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [12:49:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:49:44] Load avg. on willow is OK: OK - load average: 12.34, 14.05, 14.98 [12:57:37] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 421804 MB (7% inode=41%): [13:02:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.983887/1.95, alarm hl:np_load_avg=1.835938/2.0, alarm hl:mem_free=489.000000M/350M, alarm hl:available=1/0 [13:02:45] Load avg. on willow is WARNING: WARNING - load average: 15.23, 14.64, 14.45 [13:03:05] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.145508/1.10, alarm hl:np_load_long=0.822265/1.55, alarm hl:mem_free=16467.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.145508/1.00, alarm hl:np_load_long=0.822265/1.50, alarm hl:mem_free=16467.000000M/350M, alarm hl:available=1/0 [13:03:16] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:05:16] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:06:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.090332/1.95, alarm hl:np_load_avg=1.990723/2.0, alarm hl:mem_free=345.000000M/350M, alarm hl:available=1/0 [13:08:45] Load avg. on willow is OK: OK - load average: 13.94, 14.96, 14.76 [13:10:05] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:11:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:21:16] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:24:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:28:18] [[Template:Hardcore]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7090&rcid=9325 * Brandon Sky * (+122) (Created page with " (!!!!!!!!!!)  [[Category:Icons]]") [13:37:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:37:37] [[User talk:Brandon Sky]] ! 10https://wiki.toolserver.org/w/index.php?diff=7091&oldid=7084&rcid=9326 * Brandon Sky * (-682) () [13:41:50] [[User talk:Dispenser]] ! 10https://wiki.toolserver.org/w/index.php?diff=7092&oldid=7087&rcid=9327 * Brandon Sky * (+839) () [13:44:06] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.021484/1.00, alarm hl:np_load_long=0.765625/1.50, alarm hl:mem_free=17365.000000M/350M, alarm hl:available=1/0 [13:45:05] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:48:05] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [13:50:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:57:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.095703/1.95, alarm hl:np_load_avg=2.015137/2.0, alarm hl:mem_free=342.000000M/350M, alarm hl:available=1/0 [13:57:37] [[Special:Log/delete]] delete 10 * Dispenser * (deleted "[[02User:Kristeenlemarr5810]]": Vandalism: Link spamming) [13:58:03] [[Special:Log/block]] block 10 * Dispenser * (blocked [[02User:Kristeenlemarr5810]] with an expiry time of 1 year (account creation disabled): Spamming links to external sites: Prono link spammer) [13:58:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 421601 MB (7% inode=41%): [14:04:05] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.180664/1.10, alarm hl:np_load_long=0.788086/1.55, alarm hl:mem_free=17434.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.180664/1.00, alarm hl:np_load_long=0.788086/1.50, alarm hl:mem_free=17434.000000M/350M, alarm hl:available=1/0 [14:04:11] [[Special:Log/delete]] delete 10 * Dispenser * (deleted "[[02User:Brandon Sky/signature10]]": non-use: underiserable workaround for the signature size limit, includes deprecated HTML) [14:06:57] Load avg. on willow is WARNING: WARNING - load average: 18.02, 17.53, 16.53 [14:07:15] [[Talk:Main Page]] 10https://wiki.toolserver.org/w/index.php?diff=7093&oldid=7088&rcid=9331 * Dispenser * (+194) (/* Https red slash */ Loading CSS+images over HTTP) [14:11:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:16:32] [[Special:Log/patrol]] patrol 10 * Dispenser * (marked revision 6933 of [[02User:Maksat10]] patrolled ) [14:18:02] [[Special:Log/delete]] delete 10 * Dispenser * (deleted "[[02User:Frshven610]]": non-use: Link spamming) [14:21:06] [[Special:Log/newusers]] create 10 * Kinkreet * (New user account) [14:24:27] [[Special:Log/delete]] delete 10 * Dispenser * (deleted "[[02Venice Travel is an excellent source of advice10]]": non-use: Link spamming) [14:24:30] [[Special:Log/block]] block 10 * Dispenser * (blocked [[02User:CarruthTuel10]] with an expiry time of 3 months (account creation disabled): Spamming links to external sites) [14:24:48] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7094&oldid=7093&rcid=9336 * Kinkreet * (+227) (/* Citation Bot */ new section) [14:25:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:38:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:38:29] [[Talk:Main Page]] 10https://wiki.toolserver.org/w/index.php?diff=7095&oldid=7094&rcid=9337 * Dispenser * (+229) (/* Citation Bot */ [[Account renewal]] is a security feature) [14:42:26] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:45:34] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7096&oldid=7095&rcid=9338 * Brandon Sky * (+118) (/* Https red slash */ ) [14:48:05] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [14:49:18] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7097&oldid=7096&rcid=9339 * Brandon Sky * (-2) (/* Https red slash */ ) [14:49:50] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7098&oldid=7097&rcid=9340 * Brandon Sky * (-116) (/* Https red slash */ ) [14:50:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:54:44] [[User talk:Dispenser]] 10https://wiki.toolserver.org/w/index.php?diff=7099&oldid=7092&rcid=9341 * Dispenser * (+415) (/* Don't undo my edits */ Toolserver is German) [14:58:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 421478 MB (7% inode=41%): [15:03:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.959961/1.95, alarm hl:np_load_avg=2.034180/2.0, alarm hl:mem_free=383.000000M/350M, alarm hl:available=1/0 [15:06:47] [[User talk:Dispenser]] ! 10https://wiki.toolserver.org/w/index.php?diff=7100&oldid=7099&rcid=9342 * Brandon Sky * (+1400) (/* Undeletion request */ new section) [15:07:26] [[User talk:Dispenser]] ! 10https://wiki.toolserver.org/w/index.php?diff=7101&oldid=7100&rcid=9343 * Brandon Sky * (+0) (/* Undeletion request */ ) [15:07:56] Load avg. on willow is WARNING: WARNING - load average: 15.95, 16.43, 16.22 [15:07:56] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.355469/1.00, alarm hl:np_load_long=0.291504/1.50, alarm hl:mem_free=511.000000M/600M, alarm hl:available=1/0 [15:11:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:11:56] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [15:19:36] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:26:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:26:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.078613/1.95, alarm hl:np_load_avg=1.965820/2.0, alarm hl:mem_free=303.000000M/350M, alarm hl:available=1/0 [15:28:35] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:38:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:48:15] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [15:50:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:50:56] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.07, 23.91, 19.66 [15:51:55] Load avg. on willow is WARNING: WARNING - load average: 25.64, 23.87, 19.93 [15:52:55] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.76, 24.22, 20.30 [15:58:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 421357 MB (7% inode=41%): [16:02:56] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.304199/1.10, alarm hl:np_load_long=0.235351/1.55, alarm hl:mem_free=368.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.304199/1.00, alarm hl:np_load_long=0.235351/1.50, alarm hl:mem_free=368.000000M/600M, alarm hl:available=1/0 [16:04:56] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [16:10:36] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [16:11:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:12:55] Load avg. on willow is WARNING: WARNING - load average: 16.55, 15.57, 16.87 [16:23:45] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1941.000000 [16:26:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:36:55] Load avg. on willow is OK: OK - load average: 12.89, 14.20, 14.98 [16:38:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:45:15] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.625976/1.10, alarm hl:np_load_long=0.861328/1.55, alarm hl:mem_free=18246.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.625976/1.00, alarm hl:np_load_long=0.861328/1.50, alarm hl:mem_free=18246.000000M/600M, alarm hl:available=1/0 [16:46:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:48:15] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [16:50:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:51:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3623.000000 [16:53:55] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3560.000000 [16:58:35] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 420748 MB (7% inode=41%): [17:02:54] Load avg. on willow is WARNING: WARNING - load average: 14.40, 15.22, 14.71 [17:03:57] Load avg. on willow is OK: OK - load average: 13.55, 14.87, 14.62 [17:05:25] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:07:05] Load avg. on willow is WARNING: WARNING - load average: 14.66, 15.61, 15.02 [17:08:55] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1791.000000 [17:11:35] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [17:12:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:15:05] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:26:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:38:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:48:05] Load avg. on willow is WARNING: WARNING - load average: 14.55, 15.20, 14.65 [17:49:04] Load avg. on willow is OK: OK - load average: 13.15, 14.77, 14.54 [17:49:15] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [17:51:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:58:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 420579 MB (7% inode=41%): [18:11:34] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [18:12:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:13:05] Load avg. on willow is WARNING: WARNING - load average: 16.45, 16.20, 15.34 [18:20:05] Load avg. on willow is OK: OK - load average: 12.52, 14.30, 14.89 [18:26:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:33:05] Load avg. on willow is WARNING: WARNING - load average: 15.74, 16.31, 15.40 [18:38:05] Load avg. on willow is OK: OK - load average: 14.14, 14.86, 14.99 [18:38:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:48:35] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1966 [18:48:45] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1979.000000 [18:49:25] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [18:52:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:55:52] 3(created) [MAGNUS-315] Most wanted articles for crh wiki; Magnus' tools; Minor Improvement <10https://jira.toolserver.org/browse/MAGNUS-315> (Stephen Bruce) [18:58:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 420389 MB (7% inode=41%): [19:11:35] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [19:12:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:13:05] Load avg. on willow is WARNING: WARNING - load average: 14.21, 15.21, 14.76 [19:14:05] Load avg. on willow is OK: OK - load average: 13.60, 14.98, 14.71 [19:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [19:16:35] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3651 [19:20:25] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:23:05] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.292480/1.00, alarm hl:np_load_long=0.303223/1.50, alarm hl:mem_free=565.000000M/600M, alarm hl:available=1/0 [19:27:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:28:04] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:28:55] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [19:29:05] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [19:34:55] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:38:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:49:25] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [19:52:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:58:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 417452 MB (7% inode=41%): [20:03:15] Load avg. on willow is WARNING: WARNING - load average: 15.76, 15.49, 14.09 [20:09:15] Load avg. on willow is OK: OK - load average: 12.84, 14.68, 14.28 [20:11:35] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [20:12:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:13:14] Load avg. on willow is WARNING: WARNING - load average: 14.05, 15.21, 14.66 [20:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7202.000000 [20:17:35] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7311 [20:27:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:38:36] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:43:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.419922/1.10, alarm hl:np_load_long=0.664062/1.55, alarm hl:mem_free=17163.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.419922/1.00, alarm hl:np_load_long=0.664062/1.50, alarm hl:mem_free=17163.000000M/600M, alarm hl:available=1/0 [20:44:25] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:48:15] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure [20:49:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:49:25] SMF on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [20:49:25] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [20:49:36] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.335 second response time [20:49:36] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.323 second response time [20:50:25] SMF on wolfsbane is OK: OK - all services online [20:50:35] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.019 second response time [20:50:35] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.026 second response time [20:51:15] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure [20:52:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:58:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 418718 MB (7% inode=41%): [21:09:25] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.039062/1.00, alarm hl:np_load_long=0.784180/1.50, alarm hl:mem_free=17311.000000M/600M, alarm hl:available=1/0 [21:10:25] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:11:46] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [21:12:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:15:46] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10803.000000 [21:18:35] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10974 [21:20:25] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:27:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:34:56] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:38:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:25] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [21:51:15] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure [21:52:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:58:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 418596 MB (7% inode=41%): [22:08:15] Load avg. on willow is WARNING: WARNING - load average: 14.62, 15.35, 14.44 [22:10:14] Load avg. on willow is OK: OK - load average: 13.75, 14.69, 14.30 [22:11:46] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [22:12:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:13:15] Load avg. on willow is WARNING: WARNING - load average: 15.11, 15.24, 14.63 [22:15:46] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14402.000000 [22:18:46] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14577 [22:27:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:38:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:25] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [22:49:46] / on wolfsbane is WARNING: DISK WARNING - free space: / 6270 MB (20% inode=93%): [22:51:15] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure [22:52:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:58:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 418527 MB (7% inode=41%): [23:11:53] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure [23:12:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:13:13] Load avg. on willow is WARNING: WARNING - load average: 15.43, 14.80, 13.61 [23:14:12] Load avg. on willow is OK: OK - load average: 14.38, 14.73, 13.68 [23:15:52] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18006.000000 [23:19:43] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18237 [23:24:59] @replag [23:24:59] Thehelpfulone: s2-user: 10s [-0.00 s/s]; s3-rr-a: 24s [-0.00 s/s]; s3-user: 24s [-0.00 s/s]; s5-rr-a: 5h 9m 20s [+0.05 s/s]; s5-user: 5h 9m 20s [+0.05 s/s] [23:25:55] 3(commented) [UTRS-98] Account creation is impossible <10https://jira.toolserver.org/browse/UTRS-98> (DeltaQuad) [23:27:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:38:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:49:45] APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates). [23:49:53] / on wolfsbane is WARNING: DISK WARNING - free space: / 6039 MB (20% inode=93%): [23:52:12] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure [23:52:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:58:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 418476 MB (7% inode=41%):