[00:02:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:02:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.071777/1.95, alarm hl:np_load_avg=1.894531/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.071777/2.3, alarm hl:np_load_long=1.638184/2.5, alarm hl:cpu=72.000000/98, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [00:07:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:08:58] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:09:58] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313754 MB (5% inode=34%): [00:14:57] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:17:58] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.936035/1.95, alarm hl:np_load_avg=1.317383/2.0, alarm hl:mem_free=295.000000M/350M, alarm hl:available=1/0 [00:32:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:55:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:02:37] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:02:56] Load avg. on willow is WARNING: WARNING - load average: 12.59, 15.04, 13.85 [01:03:57] Load avg. on willow is OK: OK - load average: 11.87, 14.53, 13.75 [01:07:57] Load avg. on willow is WARNING: WARNING - load average: 14.90, 16.11, 14.64 [01:08:06] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:09:07] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:10:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315059 MB (5% inode=34%): [01:33:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:33:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.154297/1.10, alarm hl:np_load_long=0.827148/1.55, alarm hl:mem_free=12737.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.154297/1.00, alarm hl:np_load_long=0.827148/1.50, alarm hl:mem_free=12737.000000M/600M, alarm hl:available=1/0 [01:34:18] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [01:55:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:01:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.687988/1.95, alarm hl:np_load_avg=2.173828/2.0, alarm hl:mem_free=377.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.687988/2.3, alarm hl:np_load_long=1.779297/2.5, alarm hl:cpu=87.400000/98, alarm hl:mem_free=377.000000M/200M, alarm hl:available=1/0 [02:02:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:03:16] Load avg. on willow is WARNING: WARNING - load average: 13.75, 15.52, 13.95 [02:05:16] Load avg. on willow is OK: OK - load average: 12.76, 14.62, 13.80 [02:06:26] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:08:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:09:16] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:09:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.667480/1.95, alarm hl:np_load_avg=1.875000/2.0, alarm hl:mem_free=194.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.667480/2.3, alarm hl:np_load_long=1.783203/2.5, alarm hl:cpu=94.000000/98, alarm hl:mem_free=194.000000M/200M, alarm hl:available=1/0 [02:11:16] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315013 MB (5% inode=34%): [02:12:47] [[Special:Log/newusers]] create 10 * Sgolldmiss * (New user account) [02:13:47] [[User talk:Sgolldmiss]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7223&rcid=9632 * Sgolldmiss * (+150) (Created page with "Điện lạnh Sơn Tùng - nhà cung cấp các thiết bị điện lạnh uy tín ở Vinh - Nghệ An [http://dienlanhsontung.com/ dien lanh vinh]") [02:14:16] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.065430/1.00, alarm hl:np_load_long=0.874023/1.50, alarm hl:mem_free=12550.000000M/600M, alarm hl:available=1/0 [02:14:28] [[Special:Log/move]] move 10 * Sgolldmiss * (moved [[02User talk:Sgolldmiss10]] to [[Dien lanh vinh]]) [02:15:16] Load avg. on willow is WARNING: WARNING - load average: 13.84, 15.93, 15.00 [02:15:17] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:33:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:44:16] Load avg. on willow is WARNING: WARNING - load average: 15.81, 16.24, 16.04 [02:45:16] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.962891/1.10, alarm hl:np_load_long=0.967773/1.55, alarm hl:mem_free=12751.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.962891/1.00, alarm hl:np_load_long=0.967773/1.50, alarm hl:mem_free=12751.000000M/600M, alarm hl:available=1/0 [02:47:16] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:53:15] Load avg. on willow is OK: OK - load average: 12.38, 13.46, 14.94 [02:56:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:56:16] Load avg. on willow is WARNING: WARNING - load average: 14.78, 14.83, 15.26 [02:57:16] Load avg. on willow is OK: OK - load average: 13.47, 14.21, 15.00 [03:02:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:08:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:09:26] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:11:17] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313199 MB (5% inode=34%): [03:27:26] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:32:26] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:33:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:44:27] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.271484/1.10, alarm hl:np_load_long=0.845703/1.55, alarm hl:mem_free=11903.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.271484/1.00, alarm hl:np_load_long=0.845703/1.50, alarm hl:mem_free=11903.000000M/600M, alarm hl:available=1/0 [03:45:26] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:56:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:02:16] Load avg. on willow is WARNING: WARNING - load average: 15.94, 15.15, 13.79 [04:02:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:03:15] Load avg. on willow is OK: OK - load average: 13.86, 14.55, 13.66 [04:07:16] Load avg. on willow is WARNING: WARNING - load average: 18.44, 16.09, 14.41 [04:08:17] @replag all [04:08:18] matthewrbowker: s1-rr-a: 0s [-]; s1-user: 0s [-]; s2-user: 2h 8m 57s [+0.15 s/s]; s2-user-c: 2s [+0.00 s/s]; s3-rr-a: 1s [-0.00 s/s]; s3-user: 1s [-0.00 s/s]; s4-rr-a: 2s [+0.00 s/s]; s4-user: 2s [+0.00 s/s] [04:08:19] matthewrbowker: s5-rr-a: 5s [+0.00 s/s]; s5-user: 5s [+0.00 s/s]; s5-user-c: 2s [+0.00 s/s]; s6-rr-a: 5s [+0.00 s/s]; s6-user: 6s [+0.00 s/s]; s7-rr-a: 1s [-]; s7-user: 1s [-] [04:08:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:09:36] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:12:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313150 MB (5% inode=34%): [04:17:26] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:34:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:56:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:02:43] Load avg. on willow is WARNING: WARNING - load average: 17.77, 17.04, 15.11 [05:03:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:08:44] Load avg. on willow is OK: OK - load average: 10.74, 14.45, 14.60 [05:08:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:09:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:12:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313084 MB (5% inode=34%): [05:14:43] Load avg. on willow is WARNING: WARNING - load average: 19.87, 18.40, 16.22 [05:23:43] Load avg. on willow is OK: OK - load average: 8.69, 13.22, 14.98 [05:35:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:36:44] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.007812/1.10, alarm hl:np_load_long=0.994140/1.55, alarm hl:mem_free=12888.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.007812/1.00, alarm hl:np_load_long=0.994140/1.50, alarm hl:mem_free=12888.000000M/600M, alarm hl:available=1/0 [05:37:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:44:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.157227/1.95, alarm hl:np_load_avg=2.238770/2.0, alarm hl:mem_free=358.000000M/350M, alarm hl:available=1/0 [05:56:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:56:43] Load avg. on willow is WARNING: WARNING - load average: 13.43, 18.04, 18.39 [06:03:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:05:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.48, 22.63, 20.05 [06:08:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:10:53] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:12:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313050 MB (5% inode=34%): [06:15:44] Load avg. on willow is WARNING: WARNING - load average: 12.91, 18.27, 19.90 [06:35:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:38:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:41:43] Load avg. on willow is OK: OK - load average: 12.12, 13.34, 14.96 [06:43:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.682617/1.95, alarm hl:np_load_avg=2.125488/2.0, alarm hl:mem_free=212.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.682617/2.3, alarm hl:np_load_long=2.021973/2.5, alarm hl:cpu=97.000000/98, alarm hl:mem_free=212.000000M/200M, alarm hl:available=1/0 [06:44:43] Load avg. on willow is WARNING: WARNING - load average: 18.55, 16.91, 16.19 [06:52:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:56:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:59:43] Load avg. on willow is OK: OK - load average: 12.54, 13.45, 14.95 [07:02:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.989746/1.95, alarm hl:np_load_avg=2.078125/2.0, alarm hl:mem_free=300.000000M/350M, alarm hl:available=1/0 [07:03:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:08:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:10:53] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:12:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312937 MB (5% inode=34%): [07:14:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:35:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:35:53] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.297851/1.10, alarm hl:np_load_long=0.272949/1.55, alarm hl:mem_free=334.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.297851/1.00, alarm hl:np_load_long=0.272949/1.50, alarm hl:mem_free=334.000000M/600M, alarm hl:available=1/0 [07:39:52] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [07:42:33] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:42:53] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.208496/1.00, alarm hl:np_load_long=0.254395/1.50, alarm hl:mem_free=507.000000M/600M, alarm hl:available=1/0 [07:52:02] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:52:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.163086/1.10, alarm hl:np_load_long=0.732422/1.55, alarm hl:mem_free=12910.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.163086/1.00, alarm hl:np_load_long=0.732422/1.50, alarm hl:mem_free=12910.000000M/600M, alarm hl:available=1/0 [07:53:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:56:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:02:42] Load avg. on willow is WARNING: WARNING - load average: 16.53, 15.75, 14.30 [08:02:53] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2024198s failure: longrun-sol@willow in error state: QERROR as result of job 2024198s failure [08:03:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:03:42] Load avg. on willow is OK: OK - load average: 13.07, 14.91, 14.09 [08:08:43] Load avg. on willow is WARNING: WARNING - load average: 12.95, 15.38, 14.56 [08:08:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:10:53] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:12:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312851 MB (5% inode=34%): [08:35:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:40:45] 3(resolved) [ARI-7] GoblinBot4 Revert me for Leaving Message on Talk Page <10https://jira.toolserver.org/browse/ARI-7> [08:40:47] 3(reopened) [ARI-7] GoblinBot4 Revert me for Leaving Message on Talk Page <10https://jira.toolserver.org/browse/ARI-7> [08:40:48] 3(updated) [ARI-7] GoblinBot4 Revert me for Leaving Message on Talk Page <10https://jira.toolserver.org/browse/ARI-7> [08:40:48] 3(work started) [ARI-7] GoblinBot4 Revert me for Leaving Message on Talk Page <10https://jira.toolserver.org/browse/ARI-7> [08:40:48] 3(assigned) [ARI-7] GoblinBot4 Revert me for Leaving Message on Talk Page <10https://jira.toolserver.org/browse/ARI-7> [08:42:41] 3(work started) [ARI-7] GoblinBot4 Revert me for Leaving Message on Talk Page <10https://jira.toolserver.org/browse/ARI-7> [08:43:54] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.048828/1.00, alarm hl:np_load_long=0.692383/1.50, alarm hl:mem_free=12533.000000M/600M, alarm hl:available=1/0 [08:44:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:52:16] [[Database access]] ! 10https://wiki.toolserver.org/w/index.php?diff=7226&oldid=7018&rcid=9634 * Liangent * (+38) (/* namespacename */ ) [08:56:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:02:53] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2024198s failure: longrun-sol@willow in error state: QERROR as result of job 2024198s failure [09:03:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:08:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:11:53] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:12:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312746 MB (5% inode=34%): [09:30:43] 3(closed) [ARI-7] GoblinBot4 Revert me for Leaving Message on Talk Page <10https://jira.toolserver.org/browse/ARI-7> [09:35:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:56:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:03:03] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2024198s failure: longrun-sol@willow in error state: QERROR as result of job 2024198s failure [10:03:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:04:03] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.012695/1.00, alarm hl:np_load_long=0.778320/1.50, alarm hl:mem_free=12570.000000M/600M, alarm hl:available=1/0 [10:05:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:09:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:12:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:12:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312628 MB (5% inode=34%): [10:35:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:40:02] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.065430/1.00, alarm hl:np_load_long=0.785156/1.50, alarm hl:mem_free=12972.000000M/600M, alarm hl:available=1/0 [10:41:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:45:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.288086/1.10, alarm hl:np_load_long=0.858398/1.55, alarm hl:mem_free=12577.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.288086/1.00, alarm hl:np_load_long=0.858398/1.50, alarm hl:mem_free=12577.000000M/600M, alarm hl:available=1/0 [10:55:02] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.273926/1.10, alarm hl:np_load_long=0.266113/1.55, alarm hl:mem_free=431.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.273926/1.00, alarm hl:np_load_long=0.266113/1.50, alarm hl:mem_free=431.000000M/600M, alarm hl:available=1/0 [10:56:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:57:03] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [11:02:02] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.595215/1.10, alarm hl:np_load_long=0.347168/1.55, alarm hl:mem_free=453.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.595215/1.00, alarm hl:np_load_long=0.347168/1.50, alarm hl:mem_free=453.000000M/600M, alarm hl:available=1/0 [11:03:03] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2024198s failure: longrun-sol@willow in error state: QERROR as result of job 2024198s failure [11:03:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:09:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.685547/1.10, alarm hl:np_load_long=1.233399/1.55, alarm hl:mem_free=12686.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.685547/1.00, alarm hl:np_load_long=1.233399/1.50, alarm hl:mem_free=12686.000000M/600M, alarm hl:available=1/0 [11:09:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:11:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:12:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:12:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312487 MB (5% inode=34%): [11:17:03] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.095703/1.00, alarm hl:np_load_long=1.124024/1.50, alarm hl:mem_free=12849.000000M/600M, alarm hl:available=1/0 [11:24:44] 3(created) [TS-1378] Create MMP 'ac'; Toolserver: Accounts; Task <10https://jira.toolserver.org/browse/TS-1378> (Christian Thiele) [11:27:42] 3(commented) [ENWPONE-28] Page not listing unassessed articles <10https://jira.toolserver.org/browse/ENWPONE-28> (CBM) [11:35:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:35:53] Load avg. on adenia is WARNING: WARNING - load average: 15.79, 11.59, 6.61 [11:46:03] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.934570/1.10, alarm hl:np_load_long=0.563965/1.55, alarm hl:mem_free=482.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.934570/1.00, alarm hl:np_load_long=0.563965/1.50, alarm hl:mem_free=482.000000M/600M, alarm hl:available=1/0 [11:47:53] Load avg. on adenia is OK: OK - load average: 12.77, 14.84, 11.49 [11:52:03] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [11:55:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.198242/1.10, alarm hl:np_load_long=1.694336/1.55, alarm hl:mem_free=13085.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.198242/1.00, alarm hl:np_load_long=1.694336/1.50, alarm hl:mem_free=13085.000000M/600M, alarm hl:available=1/0 [11:56:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:03:03] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2024198s failure: longrun-sol@willow in error state: QERROR as result of job 2024198s failure [12:03:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:05:03] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.658203/1.10, alarm hl:np_load_long=0.624023/1.55, alarm hl:mem_free=440.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.658203/1.00, alarm hl:np_load_long=0.624023/1.50, alarm hl:mem_free=440.000000M/600M, alarm hl:available=1/0 [12:09:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:11:32] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:32] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:43] /tmp on wolfsbane is UNKNOWN: NRPE: Unable to read output [12:12:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:12:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:12:33] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 82 MB (6% inode=97%): [12:12:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312388 MB (5% inode=34%): [12:13:22] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.933 second response time [12:13:23] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.380 second response time [12:14:23] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [12:14:32] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 843 MB (41% inode=97%): [12:15:02] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane disabled: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [12:35:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:56:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:57:03] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.549316/1.10, alarm hl:np_load_long=0.423340/1.55, alarm hl:mem_free=411.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.549316/1.00, alarm hl:np_load_long=0.423340/1.50, alarm hl:mem_free=411.000000M/600M, alarm hl:available=1/0 [12:58:03] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane disabled: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [13:03:03] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.521484/1.10, alarm hl:np_load_long=0.451660/1.55, alarm hl:mem_free=227.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.521484/1.00, alarm hl:np_load_long=0.451660/1.50, alarm hl:mem_free=227.000000M/600M, alarm hl:available=1/0 [13:03:03] Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in error state: QERROR as result of job 2024198s failure: longrun-sol@willow in error state: QERROR as result of job 2024198s failure [13:03:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:07:33] / on wolfsbane is WARNING: DISK WARNING - free space: / 6270 MB (20% inode=93%): [13:09:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:11:23] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.132 second response time [13:12:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:12:22] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.016 second response time [13:12:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312230 MB (5% inode=34%): [13:35:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:56:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:59:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:03:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:07:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.198730/1.95, alarm hl:np_load_avg=1.772949/2.0, alarm hl:mem_free=655.000000M/350M, alarm hl:available=1/0 [14:07:44] / on wolfsbane is WARNING: DISK WARNING - free space: / 5906 MB (19% inode=93%): [14:08:25] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:08:25] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.255371/1.00, alarm hl:np_load_long=0.344726/1.50, alarm hl:mem_free=556.000000M/600M, alarm hl:available=1/0 [14:09:20] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane disabled: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [14:09:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:12:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:12:39] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 311908 MB (5% inode=34%): [14:20:50] Load avg. on willow is WARNING: WARNING - load average: 16.16, 15.49, 14.20 [14:21:50] Load avg. on willow is OK: OK - load average: 13.12, 14.77, 14.04 [14:26:50] Load avg. on willow is WARNING: WARNING - load average: 15.07, 15.22, 14.38 [14:36:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:43:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125000/1.10, alarm hl:np_load_long=0.788086/1.55, alarm hl:mem_free=12705.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125000/1.00, alarm hl:np_load_long=0.788086/1.50, alarm hl:mem_free=12705.000000M/600M, alarm hl:available=1/0 [14:45:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:52:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:57:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:03:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:07:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.075195/1.95, alarm hl:np_load_avg=1.803711/2.0, alarm hl:mem_free=699.000000M/350M, alarm hl:available=1/0 [15:07:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 5559 MB (18% inode=93%): [15:08:20] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:10:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:12:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 311428 MB (5% inode=34%): [15:13:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:17:30] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:30:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.897949/1.95, alarm hl:np_load_avg=1.169922/2.0, alarm hl:mem_free=234.000000M/350M, alarm hl:available=1/0 [15:31:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:32:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:36:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:47:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.116211/1.10, alarm hl:np_load_long=0.776367/1.55, alarm hl:mem_free=12624.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.116211/1.00, alarm hl:np_load_long=0.776367/1.50, alarm hl:mem_free=12624.000000M/600M, alarm hl:available=1/0 [15:48:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:57:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:03:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:07:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 5178 MB (17% inode=93%): [16:10:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:12:51] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 311334 MB (5% inode=34%): [16:13:29] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:31:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.489258/1.95, alarm hl:np_load_avg=2.086426/2.0, alarm hl:mem_free=585.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.489258/2.3, alarm hl:np_load_long=1.718750/2.5, alarm hl:cpu=83.300000/98, alarm hl:mem_free=585.000000M/200M, alarm hl:available=1/0 [16:32:50] Load avg. on willow is WARNING: WARNING - load average: 14.82, 15.70, 13.73 [16:33:20] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.334473/1.10, alarm hl:np_load_long=0.309082/1.55, alarm hl:mem_free=351.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.334473/1.00, alarm hl:np_load_long=0.309082/1.50, alarm hl:mem_free=351.000000M/600M, alarm hl:available=1/0 [16:33:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:34:50] Load avg. on willow is OK: OK - load average: 12.87, 14.76, 13.62 [16:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:37:20] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane disabled: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [16:42:30] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.273438/1.10, alarm hl:np_load_long=0.274902/1.55, alarm hl:mem_free=211.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.273438/1.00, alarm hl:np_load_long=0.274902/1.50, alarm hl:mem_free=211.000000M/600M, alarm hl:available=1/0 [16:43:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.440430/1.10, alarm hl:np_load_long=0.851562/1.55, alarm hl:mem_free=12403.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.440430/1.00, alarm hl:np_load_long=0.851562/1.50, alarm hl:mem_free=12403.000000M/600M, alarm hl:available=1/0 [16:47:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:57:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:58:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.243164/1.10, alarm hl:np_load_long=1.023438/1.55, alarm hl:mem_free=11868.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.243164/1.00, alarm hl:np_load_long=1.023438/1.50, alarm hl:mem_free=11868.000000M/600M, alarm hl:available=1/0 [17:04:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:07:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 4839 MB (16% inode=93%): [17:10:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:13:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:13:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 310696 MB (5% inode=34%): [17:15:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.498535/1.95, alarm hl:np_load_avg=1.619629/2.0, alarm hl:mem_free=306.000000M/350M, alarm hl:available=1/0 [17:17:29] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:25:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.219238/1.95, alarm hl:np_load_avg=1.857910/2.0, alarm hl:mem_free=554.000000M/350M, alarm hl:available=1/0 [17:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:57:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:04:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:07:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4487 MB (14% inode=93%): [18:10:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:13:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:14:44] 3(created) [ET-48] Newsletter ready for delivery; JamesR's Tools; Minor Task <10https://jira.toolserver.org/browse/ET-48> (Keith Dorey) [18:14:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 310107 MB (5% inode=34%): [18:35:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.111816/1.95, alarm hl:np_load_avg=1.150879/2.0, alarm hl:mem_free=188.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.111816/2.3, alarm hl:np_load_long=1.300293/2.5, alarm hl:cpu=67.800000/98, alarm hl:mem_free=188.000000M/200M, alarm hl:available=1/0 [18:35:56] any reason that user_former_groups isnt listed in the toolserver? [18:36:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:44:31] s3 is down? [18:50:40] @replag [18:50:40] Betacommand: s2-user: 8h 30m 14s [+0.43 s/s]; s3-rr-a: 1m 38s [+0.00 s/s]; s3-user: 1m 38s [+0.00 s/s] [18:50:47] mmovchin: no [18:57:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:00:28] where is central_auth [19:04:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:06:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.197266/1.95, alarm hl:np_load_avg=1.485351/2.0, alarm hl:mem_free=612.000000M/350M, alarm hl:available=1/0 [19:07:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 4171 MB (13% inode=93%): [19:10:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:10:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:13:20] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.356445/1.00, alarm hl:np_load_long=0.258301/1.50, alarm hl:mem_free=573.000000M/600M, alarm hl:available=1/0 [19:13:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:14:20] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane disabled: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [19:14:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305654 MB (5% inode=33%): [19:19:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.533691/1.95, alarm hl:np_load_avg=1.331055/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.533691/2.3, alarm hl:np_load_long=1.315918/2.5, alarm hl:cpu=67.600000/98, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [19:21:19] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.218262/1.00, alarm hl:np_load_long=0.238281/1.50, alarm hl:mem_free=583.000000M/600M, alarm hl:available=1/0 [19:32:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:36:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:37:20] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:45:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.234375/1.10, alarm hl:np_load_long=0.774414/1.55, alarm hl:mem_free=13053.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.234375/1.00, alarm hl:np_load_long=0.774414/1.50, alarm hl:mem_free=13053.000000M/600M, alarm hl:available=1/0 [19:46:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:57:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:02:29] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:07:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 3839 MB (12% inode=93%): [20:10:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:13:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:14:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 291575 MB (5% inode=32%): [20:15:43] 3(created) [SWMTBOT-48] Remove from greylist when adding to blacklist; SWMTBot; Minor Improvement <10https://jira.toolserver.org/browse/SWMTBOT-48> (MF-Warburg ) [20:20:42] 3(created) [SWMTBOT-49] Whitelisted users should not get on greylist; SWMTBot; Bug <10https://jira.toolserver.org/browse/SWMTBOT-49> (MF-Warburg ) [20:24:43] 3(created) [SWMTBOT-50] Global bot list; SWMTBot; Minor New Feature <10https://jira.toolserver.org/browse/SWMTBOT-50> (MF-Warburg ) [20:36:50] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 69 MB (3% inode=97%): [20:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:38:30] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.865 second response time [20:38:50] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.081 second response time [20:39:29] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [20:39:51] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.011 second response time [20:42:50] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 625 MB (23% inode=97%): [20:45:30] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.751953/1.10, alarm hl:np_load_long=0.828125/1.55, alarm hl:mem_free=12957.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.751953/1.00, alarm hl:np_load_long=0.828125/1.50, alarm hl:mem_free=12957.000000M/600M, alarm hl:available=1/0 [20:46:39] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:47:30] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:47:40] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [20:54:30] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 2026435s failure: medium-sol@wolfsbane in error state: QERROR as result of job 2026435s failure [20:55:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.395508/1.95, alarm hl:np_load_avg=1.612305/2.0, alarm hl:mem_free=681.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.395508/2.3, alarm hl:np_load_long=1.373047/2.5, alarm hl:cpu=85.600000/98, alarm hl:mem_free=681.000000M/200M, alarm hl:available=1/0 [20:56:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:57:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:05:33] [[Category:Documentation]] ! 10https://wiki.toolserver.org/w/index.php?diff=7227&oldid=7219&rcid=9635 * Alchimista * (-1443) (Undo revision 7219 by [[Special:Contributions/46.185.196.167|46.185.196.167]] ([[User talk:46.185.196.167|talk]])) [21:05:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:08:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 3500 MB (11% inode=93%): [21:10:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:13:30] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.371094/1.10, alarm hl:np_load_long=0.887695/1.55, alarm hl:mem_free=12325.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.371094/1.00, alarm hl:np_load_long=0.887695/1.50, alarm hl:mem_free=12325.000000M/600M, alarm hl:available=1/0 [21:13:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:14:29] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:14:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 288622 MB (5% inode=32%): [21:20:30] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.317383/1.10, alarm hl:np_load_long=0.977539/1.55, alarm hl:mem_free=12370.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.317383/1.00, alarm hl:np_load_long=0.977539/1.50, alarm hl:mem_free=12370.000000M/600M, alarm hl:available=1/0 [21:28:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.159180/1.95, alarm hl:np_load_avg=1.198242/2.0, alarm hl:mem_free=222.000000M/350M, alarm hl:available=1/0 [21:31:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:36:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:45:50] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3288 MB (10% inode=93%): [21:54:30] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 2026435s failure: medium-sol@wolfsbane in error state: QERROR as result of job 2026435s failure [21:57:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:05:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:10:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:13:40] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:14:51] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308646 MB (5% inode=34%): [22:32:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.575195/1.95, alarm hl:np_load_avg=1.674805/2.0, alarm hl:mem_free=128.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.575195/2.3, alarm hl:np_load_long=1.593750/2.5, alarm hl:cpu=76.100000/98, alarm hl:mem_free=128.000000M/200M, alarm hl:available=1/0 [22:33:40] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:36:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:42:36] crontab hasn't run on willow for hours [22:42:53] at least some of it [22:45:50] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3050 MB (10% inode=93%): [22:46:10] Krinkle: works for me [22:46:25] my midnight CEST (=46 minutes ago) cronjob ran find [22:46:36] a crontab I have on willow for ~cvn hasn't run in 81991 seconds [22:46:48] tools under ~krinkle appear to be working fine [22:46:59] it is scheduled to run every minute [22:47:59] try editing the crontab [22:48:48] (I'm not too sure about crontab internals, but maybe your crontab is not loaded in the correct folder atm, and editing it will fix that?) [22:50:13] running the script manually now, maybe something else broke [22:50:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.045410/1.95, alarm hl:np_load_avg=1.749512/2.0, alarm hl:mem_free=327.000000M/350M, alarm hl:available=1/0 [22:50:39] works fine [22:54:01] any reason that user_former_groups isnt listed in the toolserver? [22:54:28] Betacommand: Is it public on wmf? [22:54:30] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 2026435s failure: medium-sol@wolfsbane in error state: QERROR as result of job 2026435s failure [22:54:34] Is that data publicly retriable ? [22:54:39] retievable [22:54:50] Krinkle: yes/no [22:55:03] former groups also contains non-manual group changes [22:55:04] it just tracks the groups that a user used to belong to [22:55:10] so logging isn't all [22:55:16] but either way, request it [22:55:49] new stuff is hidden by default, until it is requested and found to be open [22:56:01] found to be public enough( [22:56:03] * [22:57:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:01:43] 3(created) [TS-1379] user_former_groups database view; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1379> (Betacommand) [23:05:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:10:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:12:31] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:14:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 308607 MB (5% inode=34%): [23:22:19] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:30:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.406738/1.95, alarm hl:np_load_avg=1.441406/2.0, alarm hl:mem_free=163.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.406738/2.3, alarm hl:np_load_long=1.442383/2.5, alarm hl:cpu=81.700000/98, alarm hl:mem_free=163.000000M/200M, alarm hl:available=1/0 [23:30:19] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:31:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:31:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:35:39] Sun Grid Engine execd on wolfsbane is OK: testqueue@wolfsbane disabled: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [23:46:13] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2792 MB (9% inode=93%): [23:57:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk