[00:00:27] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:01:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:07:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:13:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.615723/1.95, alarm hl:np_load_avg=1.455078/2.0, alarm hl:mem_free=139.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.615723/2.3, alarm hl:np_load_long=1.324219/2.5, alarm hl:cpu=82.200000/98, alarm hl:mem_free=139.000000M/200M, alarm hl:available=1/0 [00:13:49] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:14:25] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:18:45] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:24:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:28:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:30:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.715820/1.95, alarm hl:np_load_avg=0.936035/2.0, alarm hl:mem_free=283.000000M/350M, alarm hl:available=1/0 [00:32:26] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.919922/1.10, alarm hl:np_load_long=1.466797/1.55, alarm hl:mem_free=15650.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.919922/1.00, alarm hl:np_load_long=1.466797/1.50, alarm hl:mem_free=15650.000000M/600M, alarm hl:available=1/0 [00:33:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:33:52] [[Special:Log/newusers]] create 10 * Jeanmay1987 * (New user account) [00:34:25] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:41:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.980469/1.10, alarm hl:np_load_long=1.722656/1.55, alarm hl:mem_free=15728.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.980469/1.00, alarm hl:np_load_long=1.722656/1.50, alarm hl:mem_free=15728.000000M/600M, alarm hl:available=1/0 [00:42:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:55:06] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308229 MB (5% inode=34%): [00:55:22] [[User:XxMaverickxx1988]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7175&rcid=9546 * XxMaverickxx1988 * (+286) (Created page with "Grüß Gott Meinereiner bins zudem Interessiere mich außergewöhnlich für die Gesundheit anderer Leute und auch das Internet. Bin ein sehr aufgeweckter Junger Mann und freue mi...") [01:02:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:03:39] [[Special:Log/newusers]] create 10 * Adserve2006 * (New user account) [01:24:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:28:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:43:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:56:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 310312 MB (5% inode=34%): [02:02:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:04:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.631836/1.10, alarm hl:np_load_long=0.746094/1.55, alarm hl:mem_free=15991.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.631836/1.00, alarm hl:np_load_long=0.746094/1.50, alarm hl:mem_free=15991.000000M/600M, alarm hl:available=1/0 [02:07:31] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:25:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:27:32] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.672851/1.10, alarm hl:np_load_long=1.024414/1.55, alarm hl:mem_free=16058.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.672851/1.00, alarm hl:np_load_long=1.024414/1.50, alarm hl:mem_free=16058.000000M/600M, alarm hl:available=1/0 [02:28:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:29:32] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:32:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.103516/1.95, alarm hl:np_load_avg=1.885254/2.0, alarm hl:mem_free=431.000000M/350M, alarm hl:available=1/0 [02:33:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:34:48] [[User talk:Dab]] 10https://wiki.toolserver.org/w/index.php?diff=7176&oldid=6697&rcid=9548 * Pathoschild * (+346) (+ /* New wikis */) [02:42:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.668945/1.10, alarm hl:np_load_long=1.254883/1.55, alarm hl:mem_free=15484.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.668945/1.00, alarm hl:np_load_long=1.254883/1.50, alarm hl:mem_free=15484.000000M/600M, alarm hl:available=1/0 [02:43:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:52:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:56:11] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308482 MB (5% inode=34%): [03:02:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:02:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.128906/1.95, alarm hl:np_load_avg=1.801270/2.0, alarm hl:mem_free=992.000000M/350M, alarm hl:available=1/0 [03:03:42] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:25:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:28:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:43:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:45:44] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.156250/1.10, alarm hl:np_load_long=0.799805/1.55, alarm hl:mem_free=15142.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.156250/1.00, alarm hl:np_load_long=0.799805/1.50, alarm hl:mem_free=15142.000000M/600M, alarm hl:available=1/0 [03:46:42] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:56:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308462 MB (5% inode=34%): [04:02:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:05:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.853516/1.10, alarm hl:np_load_long=1.007812/1.55, alarm hl:mem_free=14446.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.853516/1.00, alarm hl:np_load_long=1.007812/1.50, alarm hl:mem_free=14446.000000M/600M, alarm hl:available=1/0 [04:05:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.373535/1.95, alarm hl:np_load_avg=1.852051/2.0, alarm hl:mem_free=364.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.373535/2.3, alarm hl:np_load_long=1.583984/2.5, alarm hl:cpu=98.200000/98, alarm hl:mem_free=364.000000M/200M, alarm hl:available=1/0 [04:07:42] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:12:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.821289/1.95, alarm hl:np_load_avg=2.017578/2.0, alarm hl:mem_free=678.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.821289/2.3, alarm hl:np_load_long=1.721680/2.5, alarm hl:cpu=94.700000/98, alarm hl:mem_free=678.000000M/200M, alarm hl:available=1/0 [04:15:01] Load avg. on willow is WARNING: WARNING - load average: 13.94, 15.25, 13.80 [04:17:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:17:56] 3(commented) [TS-1333] Install ack <10https://jira.toolserver.org/browse/TS-1333> (Krinkle) [04:18:01] Load avg. on willow is OK: OK - load average: 11.18, 14.39, 13.81 [04:22:46] @replag [04:22:47] Krinkle: s1-rr-a: 10m 22s [+0.01 s/s]; s1-user: 10m 22s [+0.01 s/s]; s2-user: 11m 58s [-0.23 s/s]; s3-rr-a: 40s [-0.00 s/s]; s3-user: 40s [-0.00 s/s]; s5-rr-a: 23s [+0.00 s/s]; s5-user: 23s [+0.00 s/s] [04:22:55] 3(created) [TS-1369] SSL certificate problem, bad/outdated HTTPS CA; Toolserver; Blocker Bug <10https://jira.toolserver.org/browse/TS-1369> (Krinkle) [04:25:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:27:01] Load avg. on willow is WARNING: WARNING - load average: 12.95, 15.35, 14.67 [04:29:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:43:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:43:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.515625/1.10, alarm hl:np_load_long=1.454101/1.55, alarm hl:mem_free=15177.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.515625/1.00, alarm hl:np_load_long=1.454101/1.50, alarm hl:mem_free=15177.000000M/600M, alarm hl:available=1/0 [04:47:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:56:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308396 MB (5% inode=34%): [05:02:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:18:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:20:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.160156/1.95, alarm hl:np_load_avg=1.916016/2.0, alarm hl:mem_free=456.000000M/350M, alarm hl:available=1/0 [05:21:42] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:23:42] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:25:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:29:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:32:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.227539/1.95, alarm hl:np_load_avg=2.430176/2.0, alarm hl:mem_free=738.000000M/350M, alarm hl:available=1/0 [05:33:00] Load avg. on willow is WARNING: WARNING - load average: 14.95, 18.67, 16.41 [05:43:01] Load avg. on willow is CRITICAL: CRITICAL - load average: 36.25, 21.46, 18.02 [05:43:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:44:00] Load avg. on willow is WARNING: WARNING - load average: 29.68, 22.40, 18.58 [05:44:32] [[Special:Log/newusers]] create 10 * Klirza443 * (New user account) [05:50:41] [[User:Klirza443]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7177&rcid=9550 * Klirza443 * (+358) (Created page with "There are no certain things when dating. Many relationships end for nearly no cause. Maybe they will end due to the fact getting back again togather could be so great. So We have...") [05:56:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308341 MB (5% inode=34%): [06:02:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:11:50] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.312500/1.10, alarm hl:np_load_long=0.749023/1.55, alarm hl:mem_free=15383.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.312500/1.00, alarm hl:np_load_long=0.749023/1.50, alarm hl:mem_free=15383.000000M/600M, alarm hl:available=1/0 [06:13:50] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:16:51] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.196289/1.10, alarm hl:np_load_long=0.855469/1.55, alarm hl:mem_free=15545.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.196289/1.00, alarm hl:np_load_long=0.855469/1.50, alarm hl:mem_free=15545.000000M/600M, alarm hl:available=1/0 [06:25:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:29:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:29:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:32:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.236328/1.95, alarm hl:np_load_avg=2.374512/2.0, alarm hl:mem_free=1101.000000M/350M, alarm hl:available=1/0 [06:40:50] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:43:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:56:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308253 MB (5% inode=34%): [07:02:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:13:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:02] Load avg. on willow is WARNING: WARNING - load average: 10.19, 13.31, 15.25 [07:22:01] Load avg. on willow is OK: OK - load average: 9.29, 12.68, 14.91 [07:26:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:30:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:38:20] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:43:01] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1951.000000 [07:43:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:56:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308144 MB (5% inode=34%): [08:03:03] Load avg. on willow is WARNING: WARNING - load average: 13.54, 16.45, 14.22 [08:03:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:03:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.181152/1.95, alarm hl:np_load_avg=2.172363/2.0, alarm hl:mem_free=1112.000000M/350M, alarm hl:available=1/0 [08:04:02] Load avg. on willow is OK: OK - load average: 9.50, 14.88, 13.82 [08:04:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:06:11] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:07:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:11:11] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3639.000000 [08:13:19] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [08:15:10] Load avg. on willow is WARNING: WARNING - load average: 11.31, 15.47, 14.89 [08:23:11] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.016601/1.00, alarm hl:np_load_long=0.862305/1.50, alarm hl:mem_free=15456.000000M/600M, alarm hl:available=1/0 [08:24:11] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:26:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:27:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.964844/1.95, alarm hl:np_load_avg=1.769531/2.0, alarm hl:mem_free=1344.000000M/350M, alarm hl:available=1/0 [08:30:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:41:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.295899/1.10, alarm hl:np_load_long=0.973633/1.55, alarm hl:mem_free=15611.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.295899/1.00, alarm hl:np_load_long=0.973633/1.50, alarm hl:mem_free=15611.000000M/600M, alarm hl:available=1/0 [08:43:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:53:21] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:56:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308028 MB (5% inode=34%): [09:03:21] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1986107s failure: longrun-sol@willow in error state: QERROR as result of job 1986107s failure [09:07:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:11:23] / on willow is WARNING: DISK WARNING - free space: / 21989 MB (20% inode=99%): [09:11:23] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7246.000000 [09:11:30] /tmp on willow is WARNING: DISK WARNING - free space: / 21989 MB (20% inode=99%): [09:13:21] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [09:26:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:27:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.375976/1.10, alarm hl:np_load_long=0.944336/1.55, alarm hl:mem_free=16644.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.375976/1.00, alarm hl:np_load_long=0.944336/1.50, alarm hl:mem_free=16644.000000M/600M, alarm hl:available=1/0 [09:30:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:38:21] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:43:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:56:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 307004 MB (5% inode=34%): [10:03:21] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1986107s failure: longrun-sol@willow in error state: QERROR as result of job 1986107s failure [10:07:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:11:30] /tmp on willow is WARNING: DISK WARNING - free space: / 21763 MB (20% inode=98%): [10:12:20] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10909.000000 [10:12:20] / on willow is WARNING: DISK WARNING - free space: / 21760 MB (20% inode=98%): [10:14:21] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [10:19:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.595703/1.10, alarm hl:np_load_long=1.492188/1.55, alarm hl:mem_free=16497.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.595703/1.00, alarm hl:np_load_long=1.492188/1.50, alarm hl:mem_free=16497.000000M/600M, alarm hl:available=1/0 [10:22:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:26:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:30:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:33:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.135742/1.10, alarm hl:np_load_long=1.167969/1.55, alarm hl:mem_free=16287.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.135742/1.00, alarm hl:np_load_long=1.167969/1.50, alarm hl:mem_free=16287.000000M/600M, alarm hl:available=1/0 [10:40:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:43:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:44:21] Load avg. on willow is WARNING: WARNING - load average: 15.70, 13.90, 12.63 [10:45:40] / on wolfsbane is WARNING: DISK WARNING - free space: / 6289 MB (20% inode=93%): [10:46:21] Load avg. on willow is OK: OK - load average: 14.46, 14.05, 12.84 [10:46:40] / on wolfsbane is OK: DISK OK - free space: / 6291 MB (21% inode=93%): [10:50:21] Load avg. on willow is WARNING: WARNING - load average: 17.29, 14.86, 13.37 [10:50:40] / on wolfsbane is WARNING: DISK WARNING - free space: / 6253 MB (20% inode=93%): [10:56:41] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306910 MB (5% inode=34%): [11:06:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.438965/1.95, alarm hl:np_load_avg=2.224609/2.0, alarm hl:mem_free=1320.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.438965/2.3, alarm hl:np_load_long=1.890137/2.5, alarm hl:cpu=87.600000/98, alarm hl:mem_free=1320.000000M/200M, alarm hl:available=1/0 [11:07:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:08:39] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:11:40] /tmp on willow is WARNING: DISK WARNING - free space: / 21532 MB (20% inode=98%): [11:12:21] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14510.000000 [11:12:22] / on willow is WARNING: DISK WARNING - free space: / 21530 MB (20% inode=98%): [11:14:21] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [11:19:30] [[Special:Log/newusers]] create 10 * Munawarsasoli * (New user account) [11:26:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:31:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:44:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:50:53] / on wolfsbane is WARNING: DISK WARNING - free space: / 5899 MB (19% inode=93%): [11:56:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306837 MB (5% inode=34%): [12:07:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:11:43] /tmp on willow is WARNING: DISK WARNING - free space: / 21309 MB (20% inode=98%): [12:12:23] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18112.000000 [12:12:24] / on willow is WARNING: DISK WARNING - free space: / 21306 MB (20% inode=98%): [12:14:23] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [12:20:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.436523/1.95, alarm hl:np_load_avg=2.068359/2.0, alarm hl:mem_free=1551.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.436523/2.3, alarm hl:np_load_long=1.920899/2.5, alarm hl:cpu=82.800000/98, alarm hl:mem_free=1551.000000M/200M, alarm hl:available=1/0 [12:21:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:24:31] [[Special:Log/newusers]] create 10 * Franks9998 * (New user account) [12:26:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:26:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.000977/1.95, alarm hl:np_load_avg=2.027344/2.0, alarm hl:mem_free=1715.000000M/350M, alarm hl:available=1/0 [12:29:44] @replag [12:29:44] Merlissimo: s3-rr-a: 1m 14s [-0.03 s/s]; s3-user: 1m 14s [-0.03 s/s]; s5-rr-a: 5h 19m 20s [+1.00 s/s]; s5-user: 5h 19m 20s [+1.00 s/s] [12:31:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:36:23] / on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:37:23] / on willow is WARNING: DISK WARNING - free space: / 21363 MB (20% inode=98%): [12:37:51] 3(created) [TS-1370] replication on sql-s5-user stopped; Toolserver: Databases; Critical Bug <10https://jira.toolserver.org/browse/TS-1370> (merl) [12:44:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:50:33] Load avg. on willow is WARNING: WARNING - load average: 25.73, 19.18, 17.25 [12:51:03] / on wolfsbane is WARNING: DISK WARNING - free space: / 5576 MB (18% inode=93%): [12:56:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306756 MB (5% inode=34%): [13:00:33] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.35, 18.07, 16.75 [13:01:33] Load avg. on willow is WARNING: WARNING - load average: 27.24, 20.47, 17.77 [13:07:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:11:43] /tmp on willow is WARNING: DISK WARNING - free space: / 21174 MB (20% inode=98%): [13:13:23] [[IT Certification material]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7178&rcid=9553 * IT-Certification * (+232) (Created page with "CertClick IT Certification Training Materials, include PDF document and test engine and Online Testing Account, Just 99USD for...") [13:13:33] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21776.000000 [13:14:23] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [13:20:34] Load avg. on willow is OK: OK - load average: 12.57, 13.40, 14.88 [13:21:52] 3(created) [ACCAPP-503] Running Bot on Arabic Wikipedia; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-503> (Ciphers) [13:23:34] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.149414/1.10, alarm hl:np_load_long=1.232422/1.55, alarm hl:mem_free=16283.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.149414/1.00, alarm hl:np_load_long=1.232422/1.50, alarm hl:mem_free=16283.000000M/600M, alarm hl:available=1/0 [13:26:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:27:33] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:28:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:31:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:32:34] Load avg. on willow is WARNING: WARNING - load average: 12.55, 15.66, 15.08 [13:32:53] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1987246s failure: longrun-sol@willow in error state: QERROR as result of job 1987246s failure [13:33:33] Load avg. on willow is OK: OK - load average: 10.19, 14.38, 14.66 [13:33:33] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.025390/1.10, alarm hl:np_load_long=1.329101/1.55, alarm hl:mem_free=16574.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.025390/1.00, alarm hl:np_load_long=1.329101/1.50, alarm hl:mem_free=16574.000000M/600M, alarm hl:available=1/0 [13:33:43] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:37:33] / on willow is WARNING: DISK WARNING - free space: / 21144 MB (20% inode=98%): [13:44:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:44:40] [[User:Franks9998]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7179&rcid=9554 * Franks9998 * (+419) (Created page with "My name's Rosalie Henry and I'm new to this wiki, but I aspire to be helpful ASAP. I've been unemployed for nearly 4 years now, and I have to practice my computer and editing ski...") [13:51:03] / on wolfsbane is WARNING: DISK WARNING - free space: / 5188 MB (17% inode=93%): [13:51:09] [[Special:Log/newusers]] create 10 * Maverickxx54 * (New user account) [13:56:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306645 MB (5% inode=34%): [14:07:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:08:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.055664/1.10, alarm hl:np_load_long=1.078125/1.55, alarm hl:mem_free=15857.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.055664/1.00, alarm hl:np_load_long=1.078125/1.50, alarm hl:mem_free=15857.000000M/600M, alarm hl:available=1/0 [14:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 21009 MB (20% inode=98%): [14:12:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:13:34] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25383.000000 [14:14:33] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [14:16:43] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.324707/1.10, alarm hl:np_load_long=0.654785/1.55, alarm hl:mem_free=1445.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.324707/1.00, alarm hl:np_load_long=0.654785/1.50, alarm hl:mem_free=1445.000000M/600M, alarm hl:available=1/0 [14:16:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.063477/1.10, alarm hl:np_load_long=1.324219/1.55, alarm hl:mem_free=15628.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.063477/1.00, alarm hl:np_load_long=1.324219/1.50, alarm hl:mem_free=15628.000000M/600M, alarm hl:available=1/0 [14:18:43] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [14:20:03] Load avg. on ortelius is WARNING: WARNING - load average: 23.16, 20.39, 11.29 [14:24:03] Load avg. on ortelius is OK: OK - load average: 6.81, 13.93, 10.75 [14:26:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:31:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:32:54] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1987246s failure: longrun-sol@willow in error state: QERROR as result of job 1987246s failure [14:35:52] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:37:33] / on willow is WARNING: DISK WARNING - free space: / 20911 MB (19% inode=98%): [14:44:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:51:03] / on wolfsbane is WARNING: DISK WARNING - free space: / 4834 MB (16% inode=93%): [14:56:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306530 MB (5% inode=34%): [15:08:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:09:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.432617/1.10, alarm hl:np_load_long=0.983398/1.55, alarm hl:mem_free=15858.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.432617/1.00, alarm hl:np_load_long=0.983398/1.50, alarm hl:mem_free=15858.000000M/600M, alarm hl:available=1/0 [15:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 20776 MB (19% inode=98%): [15:13:34] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28985.000000 [15:14:34] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [15:15:54] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:20:54] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.522461/1.10, alarm hl:np_load_long=1.171875/1.55, alarm hl:mem_free=16163.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.522461/1.00, alarm hl:np_load_long=1.171875/1.50, alarm hl:mem_free=16163.000000M/600M, alarm hl:available=1/0 [15:27:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:29:31] Duesen_levelup: Do you guys keep graphs of cpu, memory usage etc? [15:32:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:33:54] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1987246s failure: longrun-sol@willow in error state: QERROR as result of job 1987246s failure [15:38:33] / on willow is WARNING: DISK WARNING - free space: / 20672 MB (19% inode=98%): [15:44:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:52:04] / on wolfsbane is WARNING: DISK WARNING - free space: / 4433 MB (14% inode=93%): [15:56:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306418 MB (5% inode=34%): [16:03:56] [[Заглавная страница]] !M 10https://wiki.toolserver.org/w/index.php?diff=7180&oldid=7041&rcid=9556 * Jackie * (-6) (change expired link to the sitematrix of Wikimedia projects) [16:09:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 20542 MB (19% inode=98%): [16:13:43] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32590.000000 [16:14:43] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [16:15:52] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166016/1.10, alarm hl:np_load_long=0.808594/1.55, alarm hl:mem_free=15855.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166016/1.00, alarm hl:np_load_long=0.808594/1.50, alarm hl:mem_free=15855.000000M/600M, alarm hl:available=1/0 [16:16:44] multichill: http://munin.toolserver.org/ [16:16:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:18:16] ty Dispenser, ouch @ http://munin.toolserver.org/Login/willow/memstat.html [16:18:53] And http://munin.toolserver.org/Login/willow/cpu.html doesn't look good either [16:18:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:20:16] There were some problem while DaB was away [16:27:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:30:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.248047/1.95, alarm hl:np_load_avg=1.165039/2.0, alarm hl:mem_free=1314.000000M/350M, alarm hl:available=1/0 [16:31:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:32:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:38:43] / on willow is WARNING: DISK WARNING - free space: / 20439 MB (19% inode=98%): [16:43:53] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:44:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:53:02] / on wolfsbane is WARNING: DISK WARNING - free space: / 4054 MB (13% inode=93%): [16:56:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306313 MB (5% inode=34%): [16:57:13] hello all [16:58:24] [[Заглавная страница]] !M 10https://wiki.toolserver.org/w/index.php?diff=7181&oldid=7180&rcid=9557 * Jackie * (+68) (+short DB status (status.toolserver.org is down)) [17:01:55] @replag [17:01:55] DaBPunkt: s2-user: 28m 35s [+0.02 s/s]; s3-rr-a: 24s [-0.00 s/s]; s3-user: 24s [-0.00 s/s]; s5-rr-a: 9h 51m 31s [+1.00 s/s]; s5-user: 9h 51m 31s [+1.00 s/s] [17:02:53] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1988037s failure [17:04:21] I will look after s5 after my dinner. status.toolserver.org should be back already [17:09:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:11:53] /tmp on willow is WARNING: DISK WARNING - free space: / 20486 MB (19% inode=98%): [17:13:44] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36192.000000 [17:14:43] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [17:23:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:27:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:32:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:38:44] / on willow is WARNING: DISK WARNING - free space: / 20370 MB (19% inode=98%): [17:44:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.298828/1.10, alarm hl:np_load_long=0.758789/1.55, alarm hl:mem_free=16260.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.298828/1.00, alarm hl:np_load_long=0.758789/1.50, alarm hl:mem_free=16260.000000M/600M, alarm hl:available=1/0 [17:44:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:46:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:49:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.383789/1.10, alarm hl:np_load_long=0.982422/1.55, alarm hl:mem_free=16672.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.383789/1.00, alarm hl:np_load_long=0.982422/1.50, alarm hl:mem_free=16672.000000M/600M, alarm hl:available=1/0 [17:53:14] / on wolfsbane is WARNING: DISK WARNING - free space: / 3669 MB (12% inode=93%): [17:56:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306219 MB (5% inode=34%): [18:03:05] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1988037s failure: longrun-sol@willow in error state: QERROR as result of job 1988170s failure [18:09:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:11:54] /tmp on willow is WARNING: DISK WARNING - free space: / 20225 MB (19% inode=98%): [18:13:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39795.000000 [18:14:44] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [18:24:00] re [18:26:54] 3(assigned) [TS-1370] replication on sql-s5-user stopped <10https://jira.toolserver.org/browse/TS-1370> (DaB.) [18:26:56] 3(commented) [TS-1370] replication on sql-s5-user stopped <10https://jira.toolserver.org/browse/TS-1370> (DaB.) [18:26:57] 3(work started) [TS-1370] replication on sql-s5-user stopped <10https://jira.toolserver.org/browse/TS-1370> (DaB.) [18:27:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:31:53] 3(commented) [TS-1370] replication on sql-s5-user stopped <10https://jira.toolserver.org/browse/TS-1370> (DaB.) [18:32:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:35:54] 3(commented) [TS-1370] replication on sql-s5-user stopped <10https://jira.toolserver.org/browse/TS-1370> (DaB.) [18:38:56] / on willow is WARNING: DISK WARNING - free space: / 20104 MB (19% inode=98%): [18:43:14] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3288 MB (10% inode=93%): [18:44:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:48:55] 3(assigned) [TS-1368] Restore a script file from backup <10https://jira.toolserver.org/browse/TS-1368> (DaB.) [18:52:14] / on wolfsbane is OK: DISK OK - free space: / 7436 MB (24% inode=93%): [18:56:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 306036 MB (5% inode=34%): [19:04:05] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1988037s failure: longrun-sol@willow in error state: QERROR as result of job 1988170s failure [19:09:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:11:56] /tmp on willow is WARNING: DISK WARNING - free space: / 19959 MB (19% inode=98%): [19:13:56] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43401.000000 [19:14:56] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [19:18:55] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:27:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:32:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:38:55] / on willow is WARNING: DISK WARNING - free space: / 19839 MB (18% inode=98%): [19:44:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:53:35] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:56:56] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305908 MB (5% inode=34%): [19:59:55] / on willow is OK: DISK OK - free space: / 23800 MB (22% inode=99%): [19:59:56] /tmp on willow is OK: DISK OK - free space: / 23799 MB (22% inode=99%): [20:04:06] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1988037s failure: longrun-sol@willow in error state: QERROR as result of job 1988170s failure [20:09:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:12:13] [[Special:Log/newusers]] create 10 * Proteinshakediet * (New user account) [20:14:06] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47011.000000 [20:15:06] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [20:21:42] [[User:Proteinshakediet]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7182&rcid=9559 * Proteinshakediet * (+209) (Created page with "Be healthful on the Protein Shake Diet Plan. Discover about meal substitute shakes for fat reduction and more. My Site: [http://www.proteinshakedietplan.com/meal-replacement-s...") [20:23:00] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:Proteinshakediet10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [20:23:02] [[Special:Log/delete]] delete 10 * Valhallasw * (deleted "[[02User:Proteinshakediet10]]": content was: "Be healthful on the Protein Shake Diet Plan. Discover about meal substitute shakes for fat reduction and more. My Site: [http://www.proteinsh..." (and the only contributor was "[[Special:Contributions/Proteinshakediet|Proteinshakediet]]")) [20:23:54] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:Franks999810]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [20:24:04] [[Special:Log/delete]] delete 10 * Valhallasw * (deleted "[[02User:Franks999810]]": content was: "(spam)" (and the only contributor was "[[Special:Contributions/Franks9998|Franks9998]]")) [20:24:53] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:IT-Certification10]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [20:24:58] [[Special:Log/delete]] delete 10 * Valhallasw * (deleted "[[02IT Certification material10]]": content was: "CertClick IT Certification Training Materials, include PDF document and test engine and Onli..." (and the only contributor was "[[Special:Contributions/IT-Certification|IT-Certification]]")) [20:25:12] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:Klirza44310]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [20:25:14] [[Special:Log/delete]] delete 10 * Valhallasw * (deleted "[[02User:Klirza44310]]": content was: "There are no certain things when dating. Many relationships end for nearly no cause. Maybe they will end due to the fact getting back again togather could be ..." (and the only contributor was "[[Special:Contributions/Klirza443|Klirza443]]")) [20:25:37] [[Special:Log/delete]] delete 10 * Valhallasw * (deleted "[[02User:XxMaverickxx198810]]": content was: "Grüß Gott Meinereiner bins zudem Interessiere mich außergewöhnlich für die Gesundheit anderer Leute und auch das Internet. Bin ein sehr auf..." (and the only contributor was "[[Special:Contributions/XxMaverickxx1988|XxMaverickxx1988]]")) [20:25:42] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:XxMaverickxx198810]] with an expiry time of infinite (account creation disabled): Spamming links to external sites) [20:26:25] DaBPunkt: could we change the 'create account' button to link to jira or one of the other SUL sites to create an account, and disable account creation in mediawiki? [20:26:42] that would block all the bots without impacting humans [20:27:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:30:04] valhallasw: would be an idea [20:32:47] a questy captcha would probably stop them, too [20:33:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:44:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:50:14] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:51:06] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [20:57:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305784 MB (5% inode=33%): [21:04:15] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 1988037s failure: longrun-sol@willow in error state: QERROR as result of job 1988170s failure [21:09:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:14:05] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50611.000000 [21:15:05] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [21:27:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:33:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:35:02] MZMcBride * [Toolserver-l] Spam problem on Toolserver wiki [21:35:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:41:55] Thanks valhallasw. [21:41:57] Oh. [21:42:13] I missed scrollback in here before sending that e-mail. [21:42:17] Sorry. [21:44:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:45:55] 3(resolved) [TS-1368] Restore a script file from backup <10https://jira.toolserver.org/browse/TS-1368> (DaB.) [21:57:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305678 MB (5% inode=33%): [22:09:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:14:15] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54221.000000 [22:15:14] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [22:17:18] nacht ts [22:27:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:30:53] 3(created) [TS-1371] Add account to cvn and swmtbot groups; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1371> (Sactage) [22:33:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:44:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:53:55] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305580 MB (5% inode=33%): [23:08:25] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:09:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:13:35] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.107422/1.10, alarm hl:np_load_long=0.766602/1.55, alarm hl:mem_free=16082.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.107422/1.00, alarm hl:np_load_long=0.766602/1.50, alarm hl:mem_free=16082.000000M/600M, alarm hl:available=1/0 [23:14:15] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57821.000000 [23:15:14] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [23:16:35] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [23:27:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:44:35] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.496094/1.10, alarm hl:np_load_long=1.162109/1.55, alarm hl:mem_free=16128.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.496094/1.00, alarm hl:np_load_long=1.162109/1.50, alarm hl:mem_free=16128.000000M/600M, alarm hl:available=1/0 [23:45:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:46:35] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [23:57:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305515 MB (5% inode=33%):