[00:01:45] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.096680/1.00, alarm hl:np_load_long=0.712890/1.50, alarm hl:mem_free=17471.000000M/600M, alarm hl:available=1/0  
[00:02:44] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[00:03:13] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.21, 16.00, 14.24  
[00:12:04] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[00:12:43] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[00:16:04] <tsnag>	 s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21615.000000  
[00:19:45] <tsnag>	 MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21838  
[00:28:43] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[00:37:13] <tsnag>	 Load avg. on willow is OK: OK - load average: 12.10, 13.65, 14.89  
[00:38:53] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[00:47:13] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.68, 15.76, 15.16  
[00:49:53] <tsnag>	 / on wolfsbane is WARNING: DISK WARNING - free space: / 5878 MB (19% inode=93%):  
[00:50:45] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[00:52:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[00:52:44] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:59:03] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 418210 MB (7% inode=41%):  
[01:00:13] <tsnag>	 Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[01:01:12] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 21.69, 19.46, 17.51  
[01:12:44] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[01:13:03] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[01:14:43] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.609375/1.10, alarm hl:np_load_long=0.924805/1.55, alarm hl:mem_free=17749.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.609375/1.00, alarm hl:np_load_long=0.924805/1.50, alarm hl:mem_free=17749.000000M/600M, alarm hl:available=1/0  
[01:16:43] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[01:17:04] <tsnag>	 s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25276.000000  
[01:19:45] <tsnag>	 MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25439  
[01:28:44] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[01:39:03] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[01:50:03] <tsnag>	 / on wolfsbane is WARNING: DISK WARNING - free space: / 5694 MB (19% inode=93%):  
[01:51:43] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[01:52:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[01:53:43] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:57:43] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.042969/1.10, alarm hl:np_load_long=0.870117/1.55, alarm hl:mem_free=18391.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.042969/1.00, alarm hl:np_load_long=0.870117/1.50, alarm hl:mem_free=18391.000000M/600M, alarm hl:available=1/0  
[01:59:04] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 418185 MB (7% inode=41%):  
[01:59:44] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[02:01:14] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 22.92, 19.02, 17.64  
[02:09:44] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.124024/1.10, alarm hl:np_load_long=0.926758/1.55, alarm hl:mem_free=18125.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.124024/1.00, alarm hl:np_load_long=0.926758/1.50, alarm hl:mem_free=18125.000000M/600M, alarm hl:available=1/0  
[02:13:43] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[02:14:03] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[02:18:03] <tsnag>	 s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4916.000000  
[02:18:43] <tsnag>	 MySQL slave on cassia is OK: Uptime: 3136223  Threads: 34  Questions: 3538158137  Slow queries: 462116  Opens: 6975733  Flush tables: 2  Open tables: 16360  Queries per second avg: 1128.158 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1450  
[02:19:03] <tsnag>	 s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 337.000000  
[02:28:53] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[02:31:12] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 26.34, 22.08, 20.46  
[02:34:23] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 18.12, 20.16, 19.99  
[02:35:23] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 20.37, 20.34, 20.06  
[02:39:04] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[02:50:04] <tsnag>	 / on wolfsbane is WARNING: DISK WARNING - free space: / 5510 MB (18% inode=93%):  
[02:51:44] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[02:52:22] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[02:53:44] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:55:23] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 18.21, 17.49, 18.17  
[03:00:04] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416332 MB (7% inode=41%):  
[03:08:49] <wikirc_>	 [[Special:Log/newusers]] create 10 * Bethannlaz3 *  (New user account)
[03:13:21] <wikirc_>	 [[User:Bethannlaz3]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7102&rcid=9345 * Bethannlaz3 * (+682) (Created page with "My name is Becki, but i am a 19 year-old journalism student. I am a gamer, comic book reader, sci-fi fanatic, role player, tech lover, all the while what some would  call a “gi...")
[03:13:43] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[03:13:56] <wikirc_>	 [[Special:Log/delete]] delete 10 * Dispenser *  (deleted "[[02User:Bethannlaz310]]": Vandalism)
[03:13:59] <wikirc_>	 [[Special:Log/block]] block 10 * Dispenser *  (blocked [[02User:Bethannlaz310]] with an expiry time of 1 year (account creation disabled): Spamming links to external sites)
[03:14:14] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[03:14:51] <Dispenser>	 We've been getting quite a bit of spam recently
[03:17:17] <Dispenser>	 Where are these people coming from...
[03:24:44] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.10, alarm hl:np_load_long=0.953125/1.55, alarm hl:mem_free=17974.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.112305/1.00, alarm hl:np_load_long=0.953125/1.50, alarm hl:mem_free=17974.000000M/600M, alarm hl:available=1/0  
[03:25:09] <Dispenser>	 She Becki, Bonni, and Annabel, but probably some guy named Max Lawson
[03:25:32] <tsnag>	 RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:25:44] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[03:28:43] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.264649/1.10, alarm hl:np_load_long=1.012695/1.55, alarm hl:mem_free=17893.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.264649/1.00, alarm hl:np_load_long=1.012695/1.50, alarm hl:mem_free=17893.000000M/600M, alarm hl:available=1/0  
[03:29:45] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[03:39:04] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[03:47:16] <wikirc_>	 [[User talk:Dispenser]]  10https://wiki.toolserver.org/w/index.php?diff=7103&oldid=7101&rcid=9348 * Dispenser * (+449) (/* Undeletion request */ You've missed the point of this wiki)
[03:50:04] <tsnag>	 / on wolfsbane is WARNING: DISK WARNING - free space: / 5345 MB (17% inode=93%):  
[03:51:44] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[03:52:22] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[03:53:43] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:55:04] <tsnag>	 RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[03:55:22] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 19.89, 17.61, 17.37  
[04:00:04] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416310 MB (7% inode=41%):  
[04:00:23] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[04:01:13] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[04:01:44] <tsnag>	 /sql on cassia is WARNING: DISK WARNING - free space: /sql 129985 MB (10% inode=99%):  
[04:08:43] <tsnag>	 /sql on cassia is OK: DISK OK - free space: /sql 135748 MB (11% inode=99%):  
[04:13:43] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[04:13:44] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.205078/1.10, alarm hl:np_load_long=1.049805/1.55, alarm hl:mem_free=17702.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.205078/1.00, alarm hl:np_load_long=1.049805/1.50, alarm hl:mem_free=17702.000000M/600M, alarm hl:available=1/0  
[04:30:44] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[04:38:44] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[04:39:04] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[04:43:43] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.590820/1.10, alarm hl:np_load_long=1.159180/1.55, alarm hl:mem_free=17367.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.590820/1.00, alarm hl:np_load_long=1.159180/1.50, alarm hl:mem_free=17367.000000M/600M, alarm hl:available=1/0  
[04:49:43] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[04:50:03] <tsnag>	 / on wolfsbane is WARNING: DISK WARNING - free space: / 5215 MB (17% inode=93%):  
[04:52:23] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[04:52:43] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[04:53:44] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:55:23] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.92, 15.74, 15.67  
[05:00:03] <wikirc_>	 [[Toolserver:Sandbox]] ! 10https://wiki.toolserver.org/w/index.php?diff=7104&oldid=6978&rcid=9349 * 99.61.6.66 * (+1394) (AUTOGENITURE - Self Succession)
[05:00:04] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416278 MB (7% inode=41%):  
[05:01:13] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[05:06:53] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.579101/1.10, alarm hl:np_load_long=1.053711/1.55, alarm hl:mem_free=17610.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.579101/1.00, alarm hl:np_load_long=1.053711/1.50, alarm hl:mem_free=17610.000000M/600M, alarm hl:available=1/0  
[05:08:55] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[05:13:53] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[05:18:53] <tsnag>	 /sql on cassia is WARNING: DISK WARNING - free space: /sql 129644 MB (10% inode=99%):  
[05:26:57] <tsnag>	 /sql on cassia is OK: DISK OK - free space: /sql 131639 MB (11% inode=99%):  
[05:30:53] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[05:39:14] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[05:43:53] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.222656/1.10, alarm hl:np_load_long=0.894531/1.55, alarm hl:mem_free=17646.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.222656/1.00, alarm hl:np_load_long=0.894531/1.50, alarm hl:mem_free=17646.000000M/600M, alarm hl:available=1/0  
[05:44:54] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[05:50:05] <tsnag>	 / on wolfsbane is WARNING: DISK WARNING - free space: / 5065 MB (16% inode=93%):  
[05:52:23] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[05:52:54] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[05:53:54] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.202149/1.10, alarm hl:np_load_long=0.942383/1.55, alarm hl:mem_free=17472.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.202149/1.00, alarm hl:np_load_long=0.942383/1.50, alarm hl:mem_free=17472.000000M/600M, alarm hl:available=1/0  
[05:53:54] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:55:23] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 24.43, 18.73, 18.28  
[06:00:03] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416208 MB (7% inode=41%):  
[06:01:14] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[06:13:53] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[06:14:04] <tsnag>	 Load avg. on ortelius is WARNING: WARNING - load average: 22.91, 12.37, 7.11  
[06:16:04] <tsnag>	 Load avg. on ortelius is OK: OK - load average: 12.96, 13.32, 8.20  
[06:22:33] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 27.91, 22.86, 20.39  
[06:24:34] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.66, 20.89, 19.97  
[06:25:34] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 24.18, 21.85, 20.35  
[06:27:55] <tsnag>	 /sql on cassia is WARNING: DISK WARNING - free space: /sql 129771 MB (10% inode=99%):  
[06:30:55] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[06:33:54] <tsnag>	 /sql on cassia is OK: DISK OK - free space: /sql 131745 MB (11% inode=99%):  
[06:39:13] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[06:50:33] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 19.45, 18.24, 18.93  
[06:51:04] <tsnag>	 / on wolfsbane is WARNING: DISK WARNING - free space: / 4891 MB (16% inode=93%):  
[06:51:53] <tsnag>	 /sql on cassia is WARNING: DISK WARNING - free space: /sql 129739 MB (10% inode=99%):  
[06:52:34] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[06:53:05] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[06:54:05] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:54:54] <tsnag>	 /sql on cassia is OK: DISK OK - free space: /sql 135955 MB (11% inode=99%):  
[07:00:04] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416137 MB (7% inode=41%):  
[07:01:23] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[07:04:53] <tsnag>	 /sql on cassia is WARNING: DISK WARNING - free space: /sql 128561 MB (10% inode=99%):  
[07:14:03] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[07:16:52] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.233399/1.10, alarm hl:np_load_long=0.910156/1.55, alarm hl:mem_free=17632.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.233399/1.00, alarm hl:np_load_long=0.910156/1.50, alarm hl:mem_free=17632.000000M/600M, alarm hl:available=1/0  
[07:17:53] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[07:31:04] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[07:31:13] <tsnag>	 / on wolfsbane is OK: DISK OK - free space: / 12674 MB (42% inode=93%):  
[07:33:32] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 23.45, 22.05, 20.05  
[07:39:23] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[07:41:34] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.96, 19.64, 19.93  
[07:45:04] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.962891/1.10, alarm hl:np_load_long=0.886719/1.55, alarm hl:mem_free=17080.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.962891/1.00, alarm hl:np_load_long=0.886719/1.50, alarm hl:mem_free=17080.000000M/600M, alarm hl:available=1/0  
[07:48:03] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[07:53:32] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[07:54:03] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[07:55:04] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:00:15] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416061 MB (7% inode=41%):  
[08:01:23] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[08:14:04] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[08:26:33] <tsnag>	 Load avg. on willow is OK: OK - load average: 13.61, 14.05, 14.98  
[08:31:03] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[08:39:23] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[08:40:43] <tsnag>	 RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:50:23] <tsnag>	 RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[08:53:33] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1920028s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1920028s failure  
[08:54:13] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[08:55:13] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:57:33] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.33, 14.96, 14.61  
[08:58:33] <tsnag>	 Load avg. on willow is OK: OK - load average: 14.20, 14.70, 14.54  
[09:00:22] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416107 MB (7% inode=41%):  
[09:01:23] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1918972s failure: longrun-sol@willow in error state: QERROR as result of job 1918972s failure  
[09:02:32] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.80, 15.66, 14.98  
[09:14:24] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[09:25:57] <msgbot_>	 3(created) [TS-1360] Install OpenCV libraries and python bindings; Toolserver: Software installation; Task <10https://jira.toolserver.org/browse/TS-1360>  (drtrigon)
[09:31:24] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[09:39:33] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[09:42:26] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.930664/1.95, alarm hl:np_load_avg=1.959473/2.0, alarm hl:mem_free=242.000000M/350M, alarm hl:available=1/0  
[09:42:43] <tsnag>	 Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK  
[09:49:15] <wikirc_>	 [[Special:Log/newusers]] create 10 * Alexapooleaka *  (New user account)
[09:53:25] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[09:54:25] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[09:55:23] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:58:25] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.745117/1.95, alarm hl:np_load_avg=1.821777/2.0, alarm hl:mem_free=230.000000M/350M, alarm hl:available=1/0  
[09:58:33] <tsnag>	 Load avg. on willow is OK: OK - load average: 13.88, 14.54, 15.00  
[09:59:42] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.203125/1.10, alarm hl:np_load_long=0.183106/1.55, alarm hl:mem_free=496.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.203125/1.00, alarm hl:np_load_long=0.183106/1.50, alarm hl:mem_free=496.000000M/600M, alarm hl:available=1/0  
[10:00:26] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 415843 MB (7% inode=41%):  
[10:00:43] <tsnag>	 Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK  
[10:02:33] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 18.12, 17.09, 15.98  
[10:05:02] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[10:09:43] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[10:15:23] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[10:31:26] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[10:39:44] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[10:50:33] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[10:52:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[10:53:23] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166992/1.10, alarm hl:np_load_long=0.801758/1.55, alarm hl:mem_free=16439.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166992/1.00, alarm hl:np_load_long=0.801758/1.50, alarm hl:mem_free=16439.000000M/600M, alarm hl:available=1/0  
[10:53:43] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[10:54:23] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[10:55:23] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[10:55:24] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:00:33] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 415722 MB (7% inode=41%):  
[11:01:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.382812/1.95, alarm hl:np_load_avg=2.090332/2.0, alarm hl:mem_free=328.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.382812/2.3, alarm hl:np_load_long=2.043457/2.5, alarm hl:cpu=89.500000/98, alarm hl:mem_free=328.000000M/150M, alarm hl:available=1/0  
[11:02:33] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.39, 16.11, 16.16  
[11:04:23] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416992/1.10, alarm hl:np_load_long=0.903320/1.55, alarm hl:mem_free=18112.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416992/1.00, alarm hl:np_load_long=0.903320/1.50, alarm hl:mem_free=18112.000000M/600M, alarm hl:available=1/0  
[11:05:03] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[11:07:33] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[11:16:23] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[11:29:33] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.881836/1.95, alarm hl:np_load_avg=2.034668/2.0, alarm hl:mem_free=241.000000M/350M, alarm hl:available=1/0  
[11:31:22] <johang>	 SGE is acting really weird. my job has waited 30 mins just to start running.
[11:31:33] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[11:32:39] <valhallasw>	 johang: if there are no free slots (i.e. the server is too busy), it will stay in the queue
[11:32:51] <valhallasw>	 that's the whole point of the SGE scheduling ;-)
[11:34:33] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[11:34:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[11:35:36] <johang>	 valhallasw: right, it's just that I haven't seen this phenomenon before. are all servers that loaded right now?
[11:35:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[11:36:03] <tsnag>	 SMF on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[11:36:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[11:37:02] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[11:37:24] <tsnag>	 toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[11:37:24] <tsnag>	 toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[11:37:30] <johang>	 valhallasw: wolfsbane and ortelius are basically idle atm
[11:39:53] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[11:40:12] <tsnag>	 toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time  
[11:40:13] <tsnag>	 toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time  
[11:42:42] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.981934/1.95, alarm hl:np_load_avg=1.947266/2.0, alarm hl:mem_free=216.000000M/350M, alarm hl:available=1/0  
[11:50:05] <Dispenser>	 We're going to have kill those bots on wolfsbane and ortelius since they're chewing through much needed RAM
[11:51:31] <wikirc_>	 [[Special:Log/newusers]] create 10 * Zafante *  (New user account)
[11:53:43] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[11:55:33] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:55:33] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[11:56:02] <tsnag>	 SMF on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[11:57:03] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[11:59:43] <tsnag>	 Load avg. on willow is OK: OK - load average: 12.34, 13.67, 14.83  
[11:59:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[12:01:33] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 414079 MB (7% inode=41%):  
[12:02:03] <tsnag>	 SMF on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[12:02:44] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 14.74, 15.00, 15.22  
[12:04:47] <tsnag>	 Load avg. on willow is OK: OK - load average: 13.09, 14.44, 14.99  
[12:16:23] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[12:22:03] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[12:23:43] <tsnag>	 SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[12:24:14] <tsnag>	 SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:24:25] <tsnag>	 SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[12:24:34] <tsnag>	 SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:24:34] <tsnag>	 RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:24:34] <tsnag>	 SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[12:24:53] <tsnag>	 SMF on z-dat-s3-a is OK: OK - all services online  
[12:24:53] <tsnag>	 Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:24:54] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:24:54] <tsnag>	 / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:24:54] <tsnag>	 /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:25:03] <tsnag>	 SMF on z-dat-s6-a is OK: OK - all services online  
[12:25:03] <tsnag>	 RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[12:25:12] <tsnag>	 SMTP on z-dat-s6-a is OK: SMTP OK - 0.002 sec. response time  
[12:25:23] <tsnag>	 Load avg. on z-dat-s3-a is OK: OK - load average: 1.46, 2.14, 3.04  
[12:25:23] <tsnag>	 /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 175087 MB (18% inode=99%):  
[12:25:23] <tsnag>	 / on z-dat-s3-a is OK: DISK OK - free space: / 8447 MB (28% inode=85%):  
[12:25:23] <tsnag>	 SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[12:25:23] <tsnag>	 /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 2160 MB (99% inode=99%):  
[12:25:32] <tsnag>	 SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[12:30:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[12:31:43] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[12:31:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[12:32:03] <tsnag>	 SMF on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[12:32:55] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[12:33:03] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[12:40:03] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[12:43:22] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.535156/1.10, alarm hl:np_load_long=0.786133/1.55, alarm hl:mem_free=16451.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.535156/1.00, alarm hl:np_load_long=0.786133/1.50, alarm hl:mem_free=16451.000000M/600M, alarm hl:available=1/0  
[12:45:23] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[12:55:33] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:55:33] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[12:55:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[12:56:43] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.379395/1.95, alarm hl:np_load_avg=2.243652/2.0, alarm hl:mem_free=355.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.379395/2.3, alarm hl:np_load_long=2.123047/2.5, alarm hl:cpu=99.600000/98, alarm hl:mem_free=355.000000M/150M, alarm hl:available=1/0  
[12:56:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[12:58:43] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.23, 17.27, 16.85  
[13:01:36] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413908 MB (7% inode=41%):  
[13:02:03] <tsnag>	 SMF on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[13:03:03] <tsnag>	 SMF on wolfsbane is OK: OK - all services online  
[13:06:12] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[13:06:23] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.038086/1.00, alarm hl:np_load_long=0.775390/1.50, alarm hl:mem_free=17263.000000M/600M, alarm hl:available=1/0  
[13:07:24] <tsnag>	 MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1915  
[13:07:24] <tsnag>	 s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1920.000000  
[13:08:23] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[13:16:33] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[13:29:42] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[13:31:43] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[13:32:43] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.844238/1.95, alarm hl:np_load_avg=1.950195/2.0, alarm hl:mem_free=300.000000M/350M, alarm hl:available=1/0  
[13:35:43] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[13:40:03] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[13:43:23] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3643  
[13:43:23] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3646.000000  
[13:55:33] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[13:55:43] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:56:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[13:57:43] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.959961/1.95, alarm hl:np_load_avg=2.017578/2.0, alarm hl:mem_free=324.000000M/350M, alarm hl:available=1/0  
[13:58:54] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 14.51, 15.67, 15.98  
[14:02:34] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413754 MB (7% inode=41%):  
[14:07:03] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[14:10:42] <tsnag>	 RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:16:34] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[14:20:34] <tsnag>	 RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[14:30:30] <wikirc_>	 [[User:Brandon Sky/signature]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7105&rcid=9352 * Brandon Sky * (+572) (Created page with "[[User:Brandon Sky|'''<font color="red">B</font><font color="#FF6600">r</font><font color="#FF9933">a</font><font color="#FFCC00">n</font><font color="yellow">d</font><font color...")
[14:30:43] <tsnag>	 SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[14:30:54] <tsnag>	 Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[14:31:42] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.572754/1.95, alarm hl:np_load_avg=2.439453/2.0, alarm hl:mem_free=183.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.572754/2.3, alarm hl:np_load_long=2.343262/2.5, alarm hl:cpu=91.200000/98, alarm hl:mem_free=183.000000M/150M, alarm hl:available=1/0  
[14:31:42] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[14:32:00] <wikirc_>	 [[User talk:Dispenser]] ! 10https://wiki.toolserver.org/w/index.php?diff=7106&oldid=7103&rcid=9353 * Brandon Sky * (+502) (/* Undeletion request */ )
[14:35:43] <tsnag>	 RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:36:02] <multichill>	 Dispenser: You should just nuke his crap
[14:36:46] <Joan>	 I'm about to block and kill.
[14:38:14] <wikirc_>	 [[Special:Log/block]] block 10 * MZMcBride *  (blocked [[02User:Brandon Sky10]] with an expiry time of 1 year (account creation disabled): not here to contribute productively)
[14:39:05] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Information10]]": not needed)
[14:39:10] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Alert10]]": not needed)
[14:39:14] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Negative10]]": not needed)
[14:39:19] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Important10]]": not needed)
[14:39:22] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Serious10]]": not needed)
[14:39:25] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Explicit10]]": not needed)
[14:39:30] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Success10]]": not needed)
[14:39:43] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Warning10]]": not needed)
[14:39:48] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Category:Icons10]]": not needed)
[14:39:52] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Positive10]]": not needed)
[14:40:00] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Blocked10]]": not needed)
[14:40:13] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[14:40:13] <Joan>	 Closed the wrong window.
[14:41:06] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02User talk:Brandon Sky10]]": not needed)
[14:41:10] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02User:Brandon Sky10]]": not needed)
[14:41:39] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Unblock granted10]]": not needed)
[14:41:46] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Unblock rejected10]]": not needed)
[14:41:54] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Unblock10]]": not needed)
[14:42:00] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Bannedindef10]]": not needed)
[14:42:08] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02MediaWiki talk:Blockedtext10]]": not needed)
[14:42:32] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Hardcore10]]": not needed)
[14:42:36] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Blockedindef10]]": not needed)
[14:42:40] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02User:Brandon Sky/signature10]]": not needed)
[14:42:58] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Template:Banned10]]": not needed)
[14:43:07] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02Category:Unblock10]]": not needed)
[14:43:23] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6436  
[14:43:34] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6444.000000  
[14:43:36] <wikirc_>	 [[Special:Log/delete]] delete 10 * MZMcBride *  (deleted "[[02User talk:Gifti10]]": not needed)
[14:43:51] <Joan>	 There, done.
[14:55:43] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[14:55:44] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:56:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[14:58:53] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 19.39, 19.34, 19.25  
[15:02:33] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413616 MB (7% inode=41%):  
[15:07:04] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[15:09:03] <wikirc_>	 [[Job scheduling]] ! 10https://wiki.toolserver.org/w/index.php?diff=7107&oldid=6898&rcid=9379 * 86.9.116.155 * (+4) ([[cron]])
[15:10:54] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 25.85, 21.39, 20.27  
[15:16:33] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[15:29:54] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.02, 18.13, 19.84  
[15:30:54] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 21.48, 19.58, 20.26  
[15:31:43] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[15:31:43] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.736328/1.95, alarm hl:np_load_avg=2.482422/2.0, alarm hl:mem_free=126.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.736328/2.3, alarm hl:np_load_long=2.540527/2.5, alarm hl:cpu=98.000000/98, alarm hl:mem_free=126.000000M/150M, alarm hl:available=1/0  
[15:33:54] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.87, 18.73, 19.83  
[15:40:13] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[15:43:23] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9330  
[15:44:34] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9391.000000  
[15:56:43] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:56:43] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[15:56:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[16:01:54] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 22.82, 21.14, 20.11  
[16:02:34] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413937 MB (7% inode=41%):  
[16:07:33] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[16:13:52] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.92, 19.39, 19.96  
[16:16:33] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[16:20:42] <tsnag>	 SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[16:20:52] <tsnag>	 Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[16:21:42] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[16:21:42] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.012695/1.95, alarm hl:np_load_avg=2.195801/2.0, alarm hl:mem_free=177.000000M/350M, alarm hl:available=1/0  
[16:24:42] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[16:26:04] <ggherdov>	 Hi all. About POTY (picture of the year): here I read "The result has been [...] cross-checked with Kalan's tool", linking to http://toolserver.org/~kalan/poty2010.2.html . question: what is "kalan's tool" ?
[16:40:27] <wikirc_>	 [[Special:Log/newusers]] create 10 * Hubert76a *  (New user account)
[16:40:33] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[16:43:32] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12288  
[16:44:34] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12342.000000  
[16:46:52] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.351562/1.95, alarm hl:np_load_avg=2.300293/2.0, alarm hl:mem_free=178.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.351562/2.3, alarm hl:np_load_long=2.303711/2.5, alarm hl:cpu=98.900000/98, alarm hl:mem_free=178.000000M/150M, alarm hl:available=1/0  
[16:56:53] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[16:56:53] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:57:52] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[17:02:35] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413738 MB (7% inode=41%):  
[17:03:04] <tsnag>	 Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:03:04] <tsnag>	 RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:03:33] <tsnag>	 RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[17:03:42] <tsnag>	 Environment IPMI on hyacinth is OK: ok:  temperature ok fan ok voltage ok chassis ok  
[17:07:36] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[17:08:41] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.5% (273608 kB) free!  
[17:13:52] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 18.71, 17.58, 17.52  
[17:16:03] <tsnag>	 RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:16:35] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[17:21:53] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[17:29:52] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[17:32:52] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.126953/1.95, alarm hl:np_load_avg=2.203613/2.0, alarm hl:mem_free=316.000000M/350M, alarm hl:available=1/0  
[17:38:41] <enhydra>	 ggherdov, a script that I wrote to extract data from pages and count them
[17:39:00] <enhydra>	 ggherdov, the checking will be automated this year
[17:40:42] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[17:43:34] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15305  
[17:45:42] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15411.000000  
[17:56:52] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:56:52] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[17:58:02] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[18:00:52] <tsnag>	 SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[18:01:52] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[18:03:34] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413618 MB (7% inode=41%):  
[18:08:33] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[18:08:42] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.1% (257132 kB) free!  
[18:10:16] <ggherdov>	 enhydra: I was AFK, reading your comment now. Ok I see, your tool is about extracting data. I thought it was something like "look if this PNG image appear elsewhere on the internet", to check if the POTY are original work. Nevermind! and thank you for your answer
[18:10:55] <enhydra>	 ggherdov, no, we assume that such checks are done when pictures are selected as featured
[18:11:12] <ggherdov>	 enhydra: ok
[18:14:52] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.89, 16.80, 16.92  
[18:16:42] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[18:23:42] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.083985/1.10, alarm hl:np_load_long=0.832031/1.55, alarm hl:mem_free=16705.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.083985/1.00, alarm hl:np_load_long=0.832031/1.50, alarm hl:mem_free=16705.000000M/600M, alarm hl:available=1/0  
[18:23:52] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[18:24:42] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[18:27:53] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.998535/1.95, alarm hl:np_load_avg=2.000000/2.0, alarm hl:mem_free=242.000000M/350M, alarm hl:available=1/0  
[18:39:55] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[18:40:47] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[18:43:36] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18351  
[18:43:36] <tsnag>	 SMF on wolfsbane is OK: OK - all services online  
[18:44:16] <tsnag>	 Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK  
[18:45:46] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18465.000000  
[18:55:16] <tsnag>	 RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:55:46] <tsnag>	 RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[18:57:46] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[18:57:56] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[19:01:56] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[19:01:57] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.982422/1.95, alarm hl:np_load_avg=2.231934/2.0, alarm hl:mem_free=155.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.982422/2.3, alarm hl:np_load_long=2.056152/2.5, alarm hl:cpu=97.000000/98, alarm hl:mem_free=155.000000M/150M, alarm hl:available=1/0  
[19:02:47] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[19:02:47] <tsnag>	 toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:02:47] <tsnag>	 toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[19:03:37] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413478 MB (7% inode=41%):  
[19:06:37] <tsnag>	 toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.016 second response time  
[19:06:37] <tsnag>	 toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.010 second response time  
[19:08:46] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.1% (254820 kB) free!  
[19:09:15] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[19:14:57] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.74, 17.75, 17.09  
[19:16:46] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[19:31:56] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 45.43, 26.77, 20.52  
[19:33:01] <wikirc_>	 [[Special:Log/newusers]] create 10 * Ferrinojv1647 *  (New user account)
[19:33:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.070312/1.00, alarm hl:np_load_long=0.791992/1.50, alarm hl:mem_free=17035.000000M/600M, alarm hl:available=1/0  
[19:34:38] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[19:34:57] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.47, 22.43, 19.95  
[19:40:56] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[19:43:38] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21456  
[19:45:46] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21567.000000  
[19:51:16] <wikirc_>	 [[User:Ferrinojv1647]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7108&rcid=9382 * Ferrinojv1647 * (+367) (Created page with "My name is Darin Ballard. I haven't done a wiki project before, so this is my first! My "real job" is in nursing, but in my heart I'm an internet nerd. While my boss does a decen...")
[19:57:47] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[19:58:47] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:01:57] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[20:01:57] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.645996/1.95, alarm hl:np_load_avg=2.467285/2.0, alarm hl:mem_free=521.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.645996/2.3, alarm hl:np_load_long=2.355469/2.5, alarm hl:cpu=98.500000/98, alarm hl:mem_free=521.000000M/150M, alarm hl:available=1/0  
[20:03:39] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413343 MB (7% inode=41%):  
[20:03:39] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[20:08:45] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.5% (270248 kB) free!  
[20:09:16] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[20:16:46] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[20:23:14] <Joan>	 Is willow overloaded?
[20:26:41] <Joan>	 mzmcbride@willow:~$ ps aux | grep beria | wc -l
[20:26:41] <Joan>	 103
[20:26:51] <Joan>	 That seems like a lot.
[20:26:56] <Joan>	 And load seems high.
[20:28:57] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 24.36, 21.68, 20.38  
[20:31:26] <tsnag>	 SMF on willow is OK: OK - all services online  
[20:31:26] <tsnag>	 Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[20:32:06] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.248535/1.95, alarm hl:np_load_avg=2.799316/2.0, alarm hl:mem_free=161.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.248535/2.3, alarm hl:np_load_long=2.600586/2.5, alarm hl:cpu=93.500000/98, alarm hl:mem_free=161.000000M/150M, alarm hl:available=1/0  
[20:33:56] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[20:39:57] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.64, 18.93, 19.79  
[20:40:57] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 21.25, 19.98, 20.11  
[20:41:06] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[20:44:37] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24573  
[20:46:15] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[20:46:46] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24686.000000  
[20:47:16] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[20:57:16] <tsnag>	 Sun Grid Engine execd on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[20:57:57] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[20:58:48] <tsnag>	 toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.763 second response time  
[20:58:57] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[20:59:46] <tsnag>	 toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.006 second response time  
[21:00:57] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 29.26, 20.82, 19.20  
[21:03:46] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[21:04:38] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413231 MB (7% inode=41%):  
[21:05:56] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 21.93, 21.48, 20.01  
[21:06:56] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 19.96, 21.03, 19.95  
[21:08:46] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.7% (279264 kB) free!  
[21:10:56] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 24.76, 21.24, 20.16  
[21:16:46] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[21:20:15] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[21:23:56] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 16.20, 18.92, 19.77  
[21:31:16] <tsnag>	 Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[21:31:56] <tsnag>	 Load avg. on willow is CRITICAL: CRITICAL - load average: 38.95, 23.90, 21.00  
[21:31:57] <shantanoo>	 hi
[21:32:05] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.828613/1.95, alarm hl:np_load_avg=2.985352/2.0, alarm hl:mem_free=178.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.828613/2.3, alarm hl:np_load_long=2.624512/2.5, alarm hl:cpu=94.500000/98, alarm hl:mem_free=178.000000M/150M, alarm hl:available=1/0  
[21:32:20] <shantanoo>	 getting '-bash: fork: Not enough space' on willow. anyone else facing similar issue
[21:32:54] <shantanoo>	 ' -bash: xmalloc: cannot allocate 22 bytes (262456 bytes allocated)' got this also
[21:34:06] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[21:41:06] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[21:44:37] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27565  
[21:46:46] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27676.000000  
[21:51:56] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 17.40, 18.68, 19.21  
[21:58:06] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[21:59:56] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[22:01:47] <tsnag>	 SMF on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[22:03:47] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[22:05:37] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413140 MB (7% inode=41%):  
[22:08:47] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.6% (277564 kB) free!  
[22:16:46] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[22:20:16] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[22:30:17] <tsnag>	 Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[22:31:15] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.327148/1.95, alarm hl:np_load_avg=2.219727/2.0, alarm hl:mem_free=212.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.327148/2.3, alarm hl:np_load_long=2.257324/2.5, alarm hl:cpu=97.000000/98, alarm hl:mem_free=212.000000M/150M, alarm hl:available=1/0  
[22:34:06] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[22:41:06] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[22:45:36] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30534  
[22:47:45] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30643.000000  
[22:51:57] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 15.89, 16.48, 16.96  
[22:58:15] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).  
[22:59:56] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[23:00:26] <tsnag>	 Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[23:00:27] <tsnag>	 SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon.  Check the remote server logs for error messages.  
[23:01:26] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.437988/1.95, alarm hl:np_load_avg=2.084473/2.0, alarm hl:mem_free=357.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.437988/2.3, alarm hl:np_load_long=2.081055/2.5, alarm hl:cpu=92.100000/98, alarm hl:mem_free=357.000000M/150M, alarm hl:available=1/0  
[23:01:26] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[23:03:46] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[23:05:36] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 413016 MB (7% inode=41%):  
[23:08:47] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.8% (285964 kB) free!  
[23:17:46] <tsnag>	 SMF on damiana is CRITICAL: ERROR - maintenance:  svc:/network/ldap/client:default  
[23:20:26] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[23:41:13] <tsnag>	 Free Memory on damiana is OK: OK - 7.2% (300656 kB) free.  
[23:41:21] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[23:44:11] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.7% (279944 kB) free!  
[23:46:11] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33527  
[23:48:12] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33626.000000  
[23:50:20] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[23:52:12] <tsnag>	 Free Memory on damiana is OK: OK - 7.1% (295084 kB) free.  
[23:52:12] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 13.72, 15.36, 16.40  
[23:58:21] <tsnag>	 APT on yarrow is CRITICAL: APT CRITICAL: 9 packages available for upgrade (9 critical updates).