[00:02:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.007324/1.95, alarm hl:np_load_avg=1.360351/2.0, alarm hl:mem_free=270.000000M/350M, alarm hl:available=1/0 [00:03:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:05:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:07:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:07:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.086914/1.95, alarm hl:np_load_avg=1.176758/2.0, alarm hl:mem_free=186.000000M/350M, alarm hl:available=1/0 [00:18:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391683 MB (7% inode=39%): [00:29:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:29:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:37:12] / on wolfsbane is WARNING: DISK WARNING - free space: / 6272 MB (20% inode=93%): [00:50:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:02:51] Sun Grid Engine execd on willow is WARNING: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.702637/2.3, alarm hl:np_load_long=1.404785/2.5, alarm hl:cpu=98.800000/98, alarm hl:mem_free=406.000000M/150M, alarm hl:available=1/0 [01:03:52] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:04:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.105469/1.10, alarm hl:np_load_long=0.738281/1.55, alarm hl:mem_free=17990.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.105469/1.00, alarm hl:np_load_long=0.738281/1.50, alarm hl:mem_free=17990.000000M/600M, alarm hl:available=1/0 [01:05:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [01:05:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:07:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:10:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.001465/1.95, alarm hl:np_load_avg=1.638184/2.0, alarm hl:mem_free=225.000000M/350M, alarm hl:available=1/0 [01:18:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 393998 MB (7% inode=40%): [01:29:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:38:11] / on wolfsbane is WARNING: DISK WARNING - free space: / 6084 MB (20% inode=93%): [01:50:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:05:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:07:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:07:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.131836/1.95, alarm hl:np_load_avg=1.330078/2.0, alarm hl:mem_free=302.000000M/350M, alarm hl:available=1/0 [02:09:05] @replag [02:09:06] matthewrbowker: s3-rr-a: 41s [+0.00 s/s]; s3-user: 41s [+0.00 s/s] [02:18:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 393480 MB (7% inode=40%): [02:18:50] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:29:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:35:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.744629/1.95, alarm hl:np_load_avg=1.546387/2.0, alarm hl:mem_free=218.000000M/350M, alarm hl:available=1/0 [02:38:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5950 MB (19% inode=93%): [02:48:50] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:50:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:05:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:06:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.600586/1.95, alarm hl:np_load_avg=1.544434/2.0, alarm hl:mem_free=317.000000M/350M, alarm hl:available=1/0 [03:07:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:07:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:13:11] Load avg. on willow is WARNING: WARNING - load average: 16.71, 16.03, 13.41 [03:18:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391962 MB (7% inode=39%): [03:18:12] Load avg. on willow is OK: OK - load average: 11.53, 14.61, 13.63 [03:29:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:38:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 5841 MB (19% inode=93%): [03:47:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:50:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:04:21] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.097656/1.00, alarm hl:np_load_long=0.736328/1.50, alarm hl:mem_free=17075.000000M/600M, alarm hl:available=1/0 [04:05:24] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:05:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:06:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.256348/1.95, alarm hl:np_load_avg=1.434570/2.0, alarm hl:mem_free=151.000000M/350M, alarm hl:available=1/0 [04:07:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:08:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:12:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.464355/1.95, alarm hl:np_load_avg=1.441406/2.0, alarm hl:mem_free=188.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.464355/2.3, alarm hl:np_load_long=1.437500/2.5, alarm hl:cpu=98.400000/98, alarm hl:mem_free=188.000000M/150M, alarm hl:available=1/0 [04:14:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:18:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391931 MB (7% inode=39%): [04:26:41] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:29:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:37:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.096191/1.95, alarm hl:np_load_avg=1.289062/2.0, alarm hl:mem_free=145.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.096191/2.3, alarm hl:np_load_long=1.345703/2.5, alarm hl:cpu=85.900000/98, alarm hl:mem_free=145.000000M/150M, alarm hl:available=1/0 [04:38:32] / on wolfsbane is WARNING: DISK WARNING - free space: / 5731 MB (19% inode=93%): [04:42:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:47:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.423340/1.95, alarm hl:np_load_avg=1.410156/2.0, alarm hl:mem_free=225.000000M/350M, alarm hl:available=1/0 [04:50:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:57:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:05:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:07:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:19:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391913 MB (7% inode=39%): [05:29:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:38:31] / on wolfsbane is WARNING: DISK WARNING - free space: / 5567 MB (18% inode=93%): [05:50:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:56:24] @replag [05:56:24] Joan: s3-rr-a: 1m 16s [+0.00 s/s]; s3-user: 1m 16s [+0.00 s/s]; s6-rr-a: 11s [+0.00 s/s]; s6-user: 11s [+0.00 s/s]; s7-rr-a: 11s [+0.00 s/s]; s7-user: 11s [+0.00 s/s] [06:00:10] Load avg. on willow is WARNING: WARNING - load average: 18.02, 14.37, 12.59 [06:05:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:07:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:09:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.942383/1.95, alarm hl:np_load_avg=2.035156/2.0, alarm hl:mem_free=313.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.942383/2.3, alarm hl:np_load_long=1.872559/2.5, alarm hl:cpu=99.200000/98, alarm hl:mem_free=313.000000M/150M, alarm hl:available=1/0 [06:14:20] Load avg. on willow is OK: OK - load average: 13.50, 15.00, 14.82 [06:17:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:19:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391845 MB (7% inode=39%): [06:25:32] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:21] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:29:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:32:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:37:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.382324/1.95, alarm hl:np_load_avg=1.368652/2.0, alarm hl:mem_free=282.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.382324/2.3, alarm hl:np_load_long=1.463867/2.5, alarm hl:cpu=98.900000/98, alarm hl:mem_free=282.000000M/150M, alarm hl:available=1/0 [06:38:32] / on wolfsbane is WARNING: DISK WARNING - free space: / 5340 MB (17% inode=93%): [06:38:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:46:02] [[Special:Log/newusers]] create 10 * Meetcheese123 * (New user account) [06:47:10] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [06:50:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:57:41] [[User:Meetcheese123]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7117&rcid=9409 * Meetcheese123 * (+104) (Created page with "I am a parent that loves fashion My page: [http://www.shoesforlittlefeet.com/chooze kids chooze shoes]") [07:05:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:07:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:08:30] / on wolfsbane is OK: DISK OK - free space: / 9318 MB (31% inode=93%): [07:12:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.367188/1.95, alarm hl:np_load_avg=1.298828/2.0, alarm hl:mem_free=217.000000M/350M, alarm hl:available=1/0 [07:17:36] hi [07:18:36] I am wondering if one can code a tool that assists with review of sub pages [07:19:18] I want to for example have a tool that checks sub pages of "Wikipedia/Featured_Picture_Candidates/[Ii]mage:*" [07:19:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391848 MB (7% inode=39%): [07:19:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:20:06] tool should also display how many there are [07:22:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.028320/1.95, alarm hl:np_load_avg=1.179199/2.0, alarm hl:mem_free=348.000000M/350M, alarm hl:available=1/0 [07:29:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:50:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:05:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:08:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:09:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.079590/1.95, alarm hl:np_load_avg=1.263672/2.0, alarm hl:mem_free=225.000000M/350M, alarm hl:available=1/0 [08:14:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:17:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:21] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391704 MB (7% inode=39%): [08:23:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.019043/1.95, alarm hl:np_load_avg=1.173828/2.0, alarm hl:mem_free=289.000000M/350M, alarm hl:available=1/0 [08:24:52] 3(commented) [OSM-11] WIWOSM https <10https://jira.toolserver.org/browse/OSM-11> (Marlen Caemmerer) [08:27:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:29:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:37:11] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:51:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:52:10] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:00:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.913086/1.95, alarm hl:np_load_avg=1.031738/2.0, alarm hl:mem_free=232.000000M/350M, alarm hl:available=1/0 [09:00:55] 3(created) [MAGNUS-317] Searching for talk pages of articles in category with CatScan2; Magnus' tools; New Feature <10https://jira.toolserver.org/browse/MAGNUS-317> ([[:pl:User:Skalee]]) [09:04:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:05:55] 3(updated) [MAGNUS-317] Searching for talk pages of articles in category with CatScan 2 <10https://jira.toolserver.org/browse/MAGNUS-317> ([[:pl:User:Skalee]]) [09:06:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:08:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.996094/1.95, alarm hl:np_load_avg=1.168945/2.0, alarm hl:mem_free=311.000000M/350M, alarm hl:available=1/0 [09:08:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:19:21] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391567 MB (7% inode=39%): [09:29:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:41:20] /sql on cassia is WARNING: DISK WARNING - free space: /sql 125073 MB (10% inode=99%): [09:51:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:53:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.029297/1.95, alarm hl:np_load_avg=1.199707/2.0, alarm hl:mem_free=193.000000M/350M, alarm hl:available=1/0 [10:06:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:08:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:13:10] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:10] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:14:01] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:14:01] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:19:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391494 MB (7% inode=39%): [10:22:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:29:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:31:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.345215/1.95, alarm hl:np_load_avg=1.552246/2.0, alarm hl:mem_free=258.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.345215/2.3, alarm hl:np_load_long=1.400879/2.5, alarm hl:cpu=81.600000/98, alarm hl:mem_free=258.000000M/150M, alarm hl:available=1/0 [10:32:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:32:11] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:32] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:32:33] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:32:34] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:42] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:42] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:42] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:42] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:42] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:43] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:51] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:03] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:03] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:03] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:03] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:03] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:03] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:03] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:04] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:04] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:05] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:11] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:11] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:11] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:11] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:12] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:21] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:33:32] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:33:41] Load avg. on z-dat-s3-a is OK: OK - load average: 0.23, 1.05, 1.40 [10:33:42] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 173842 MB (17% inode=99%): [10:33:42] Load avg. on z-dat-s6-a is OK: OK - load average: 0.23, 1.02, 1.39 [10:33:42] / on z-dat-s3-a is OK: DISK OK - free space: / 8434 MB (28% inode=85%): [10:33:42] Load avg. on z-dat-s7-a is OK: OK - load average: 0.23, 1.02, 1.39 [10:33:42] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 173842 MB (17% inode=99%): [10:33:43] Load avg. on z-dat-s4-a is OK: OK - load average: 0.23, 1.02, 1.39 [10:33:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 86923 MB (21% inode=99%): [10:33:44] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:33:44] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:33:45] SMF on z-dat-s7-a is OK: OK - all services online [10:33:45] SMF on z-dat-s6-a is OK: OK - all services online [10:33:46] SMF on z-dat-s4-a is OK: OK - all services online [10:33:46] SMF on z-dat-s3-a is OK: OK - all services online [10:33:51] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:33:51] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 101425 MB (25% inode=99%): [10:33:51] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2181 MB (99% inode=99%): [10:33:51] / on z-dat-s7-a is OK: DISK OK - free space: / 8434 MB (28% inode=85%): [10:33:51] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 232.000000 [10:33:52] MySQL on z-dat-s6-a is OK: Uptime: 1696395 Threads: 11 Questions: 392256607 Slow queries: 100694 Opens: 4273549 Flush tables: 2 Open tables: 2940 Queries per second avg: 231.229 [10:34:10] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [10:34:20] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [10:34:32] SMTP on z-dat-s6-a is OK: SMTP OK - 0.004 sec. response time [10:34:32] SMTP on z-dat-s3-a is OK: SMTP OK - 0.003 sec. response time [10:34:33] SMTP on z-dat-s7-a is OK: SMTP OK - 0.004 sec. response time [10:34:33] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:34:33] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:34:33] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:51:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.563965/1.95, alarm hl:np_load_avg=1.424316/2.0, alarm hl:mem_free=132.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.563965/2.3, alarm hl:np_load_long=1.317871/2.5, alarm hl:cpu=92.700000/98, alarm hl:mem_free=132.000000M/150M, alarm hl:available=1/0 [10:51:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:55:31] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:57:11] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:03:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.200195/1.10, alarm hl:np_load_long=0.751953/1.55, alarm hl:mem_free=18122.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.200195/1.00, alarm hl:np_load_long=0.751953/1.50, alarm hl:mem_free=18122.000000M/600M, alarm hl:available=1/0 [11:05:42] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:06:12] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:13] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:32] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:06:32] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:06:32] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:07:01] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:07:01] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:07:11] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [11:07:20] SMTP on z-dat-s4-a is OK: SMTP OK - 0.004 sec. response time [11:08:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:10:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.191406/1.10, alarm hl:np_load_long=0.880860/1.55, alarm hl:mem_free=17786.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.191406/1.00, alarm hl:np_load_long=0.880860/1.50, alarm hl:mem_free=17786.000000M/600M, alarm hl:available=1/0 [11:12:41] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.083496/1.00, alarm hl:np_load_long=0.562500/1.50, alarm hl:mem_free=1688.000000M/600M, alarm hl:available=1/0 [11:13:10] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:32] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:41] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [11:18:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:20:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391496 MB (7% inode=39%): [11:27:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:30:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.282226/1.10, alarm hl:np_load_long=1.185547/1.55, alarm hl:mem_free=18633.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.282226/1.00, alarm hl:np_load_long=1.185547/1.50, alarm hl:mem_free=18633.000000M/600M, alarm hl:available=1/0 [11:30:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:41:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:42:11] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:11] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:11] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:31] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:31] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:41] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:41] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:41] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:41] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:41] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:42] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:42] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:43] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:43] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:44] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:44] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:50] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:02] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:02] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:02] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:02] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:02] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:02] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:02] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:03] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:03] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:04] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:12] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:12] / on z-dat-s6-a is OK: DISK OK - free space: / 8436 MB (28% inode=85%): [11:43:12] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 2076 MB (99% inode=99%): [11:43:12] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2077 MB (99% inode=99%): [11:43:12] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 2075 MB (99% inode=99%): [11:43:12] / on z-dat-s4-a is OK: DISK OK - free space: / 8436 MB (28% inode=85%): [11:43:21] Load avg. on z-dat-s3-a is OK: OK - load average: 0.38, 1.27, 1.64 [11:43:32] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [11:43:32] Load avg. on z-dat-s6-a is OK: OK - load average: 0.32, 1.21, 1.62 [11:43:32] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 82680 MB (20% inode=99%): [11:43:32] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 173913 MB (17% inode=99%): [11:43:32] Load avg. on z-dat-s4-a is OK: OK - load average: 0.32, 1.21, 1.62 [11:43:32] Load avg. on z-dat-s7-a is OK: OK - load average: 0.32, 1.21, 1.62 [11:43:32] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 101370 MB (25% inode=99%): [11:43:33] / on z-dat-s7-a is OK: DISK OK - free space: / 8436 MB (28% inode=85%): [11:43:33] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 173913 MB (17% inode=99%): [11:43:34] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2106 MB (99% inode=99%): [11:43:34] / on z-dat-s3-a is OK: DISK OK - free space: / 8436 MB (28% inode=85%): [11:43:41] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [11:43:41] SMTP on z-dat-s3-a is OK: SMTP OK - 6.696 sec. response time [11:43:41] SMTP on z-dat-s7-a is OK: SMTP OK - 6.778 sec. response time [11:43:41] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:43:42] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 185.000000 [11:43:42] SMTP on z-dat-s6-a is OK: SMTP OK - 7.034 sec. response time [11:43:42] SMF on z-dat-s7-a is OK: OK - all services online [11:43:42] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:44:01] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:44:01] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:44:01] SMTP on hyacinth is OK: SMTP OK - 0.003 sec. response time [11:44:01] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:44:20] SMTP on z-dat-s4-a is OK: SMTP OK - 0.002 sec. response time [11:51:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:01:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.146973/1.95, alarm hl:np_load_avg=1.756836/2.0, alarm hl:mem_free=171.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.146973/2.3, alarm hl:np_load_long=1.465820/2.5, alarm hl:cpu=97.200000/98, alarm hl:mem_free=171.000000M/150M, alarm hl:available=1/0 [12:06:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:08:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:13:11] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:31] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:14:00] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:14:01] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:14:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:19:41] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.016601/1.00, alarm hl:np_load_long=1.193359/1.50, alarm hl:mem_free=18762.000000M/600M, alarm hl:available=1/0 [12:20:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391507 MB (7% inode=39%): [12:20:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:30:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:42:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.054688/1.95, alarm hl:np_load_avg=1.085938/2.0, alarm hl:mem_free=269.000000M/350M, alarm hl:available=1/0 [12:45:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:48:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.308105/1.95, alarm hl:np_load_avg=1.260742/2.0, alarm hl:mem_free=341.000000M/350M, alarm hl:available=1/0 [12:51:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:54:53] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:53] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:12] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:12] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:12] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:12] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:12] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:12] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:21] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:21] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:30] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:31] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:41] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:55:42] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:55:51] / on z-dat-s7-a is OK: DISK OK - free space: / 8434 MB (28% inode=85%): [12:55:51] /tmp on hyacinth is OK: DISK OK - free space: /tmp 1953 MB (99% inode=99%): [12:55:51] / on hyacinth is OK: DISK OK - free space: / 8434 MB (28% inode=85%): [12:55:51] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1964 MB (99% inode=99%): [12:55:51] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 101198 MB (25% inode=99%): [12:56:01] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:56:01] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:56:10] SMTP on hyacinth is OK: SMTP OK - 0.003 sec. response time [12:56:11] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:56:11] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [12:58:35] is there a special reason to stay in python 2.71? [13:06:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:08:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:20:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391379 MB (7% inode=39%): [13:30:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:43:52] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.069336/1.00, alarm hl:np_load_long=0.675781/1.50, alarm hl:mem_free=17188.000000M/600M, alarm hl:available=1/0 [13:44:51] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:51:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:07:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:09:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:18:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.907227/1.95, alarm hl:np_load_avg=1.378418/2.0, alarm hl:mem_free=239.000000M/350M, alarm hl:available=1/0 [14:20:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:21:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 390886 MB (7% inode=39%): [14:25:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.054199/1.95, alarm hl:np_load_avg=1.166016/2.0, alarm hl:mem_free=286.000000M/350M, alarm hl:available=1/0 [14:30:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:34:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:51:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:07:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:09:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:12:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.420410/1.95, alarm hl:np_load_avg=1.398926/2.0, alarm hl:mem_free=198.000000M/350M, alarm hl:available=1/0 [15:14:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:21:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391102 MB (7% inode=39%): [15:27:55] 3(commented) [ACCAPP-460] I want run a robot for fawiki, and im want mostly to faster doing job. <10https://jira.toolserver.org/browse/ACCAPP-460> (Mahdi haji) [15:31:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:48:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.315430/1.95, alarm hl:np_load_avg=1.286621/2.0, alarm hl:mem_free=208.000000M/350M, alarm hl:available=1/0 [15:50:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:51:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:07:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:08:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:09:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:12:10] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:21:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391438 MB (7% inode=39%): [16:31:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:41:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 6263 MB (20% inode=93%): [16:49:42] chicocvenancio: lack of admin time I suppose [16:51:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:56:52] [[Special:Log/newusers]] create 10 * Reinigung1 * (New user account) [16:58:26] Dispenser: that makes perfect sense [16:59:14] I managed to fix the problems in my script... have no clue if it was related to the 2.7.2 v 2.7.1 or windows v unix diferences from my machine [16:59:53] anyway, its running now, and the code was long overdue a rewrite anyway... [17:09:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:10:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:14:26] @replag [17:14:26] matthewrbowker: s2-user: 28s [+0.00 s/s]; s3-rr-a: 13s [-0.00 s/s] [17:17:55] [[User:Reinigung1]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7118&rcid=9411 * Reinigung1 * (+93) (Created page with "hi My Website: [http://www.swissoffer.ch/reinigung Kostenlose Offerten für eine Reinigung]") [17:22:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391349 MB (7% inode=39%): [17:32:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:33:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.287109/1.95, alarm hl:np_load_avg=1.318848/2.0, alarm hl:mem_free=285.000000M/350M, alarm hl:available=1/0 [17:40:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:42:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 5897 MB (19% inode=93%): [17:51:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:04:30] [[Special:Log/delete]] delete 10 * Valhallasw * (deleted "[[02User:Reinigung110]]": content was: "hi My Website: [http://www.swissoffer.ch/reinigung Kostenlose Offerten für eine Reinigung]" (and the only contributor was "[[Special:Contributions/Reinigung1|Reinigung1]]")) [18:04:44] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:Reinigung110]] with an expiry time of 1 year (account creation disabled): Spamming links to external sites) [18:08:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.107910/1.95, alarm hl:np_load_avg=1.161621/2.0, alarm hl:mem_free=198.000000M/350M, alarm hl:available=1/0 [18:09:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:10:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:11:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:13:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.365234/1.95, alarm hl:np_load_avg=1.333984/2.0, alarm hl:mem_free=234.000000M/350M, alarm hl:available=1/0 [18:23:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391280 MB (7% inode=39%): [18:32:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:38:52] / on wolfsbane is OK: DISK OK - free space: / 8601 MB (28% inode=93%): [18:47:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:51:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:52:01] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:02:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.692383/1.95, alarm hl:np_load_avg=1.452637/2.0, alarm hl:mem_free=137.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.692383/2.3, alarm hl:np_load_long=1.297851/2.5, alarm hl:cpu=82.000000/98, alarm hl:mem_free=137.000000M/150M, alarm hl:available=1/0 [19:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:11:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:16:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:21:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.134277/1.95, alarm hl:np_load_avg=1.265137/2.0, alarm hl:mem_free=133.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.134277/2.3, alarm hl:np_load_long=1.394531/2.5, alarm hl:cpu=82.600000/98, alarm hl:mem_free=133.000000M/150M, alarm hl:available=1/0 [19:24:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 383415 MB (7% inode=39%): [19:33:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:39:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:50:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36012 MB (8% inode=99%): [19:51:03] @replag [19:51:03] matthewrbowker: s3-rr-a: 46s [+0.00 s/s]; s3-user: 46s [-0.00 s/s] [19:51:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:53:01] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23485 MB (5% inode=99%): [19:54:02] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 80587 MB (19% inode=99%): [19:57:12] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:12] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:22] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:22] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:31] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:31] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:31] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:31] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:31] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:31] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:31] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:32] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:32] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:33] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:33] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:41] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:41] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:50] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:51] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:58:01] Load avg. on z-dat-s7-a is OK: OK - load average: 3.80, 7.17, 7.73 [19:58:01] /tmp on hyacinth is OK: DISK OK - free space: /tmp 1791 MB (99% inode=99%): [19:58:02] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:58:02] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:58:02] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 101627 MB (25% inode=99%): [19:58:02] / on z-dat-s7-a is OK: DISK OK - free space: / 8430 MB (28% inode=85%): [19:58:02] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1834 MB (99% inode=99%): [19:58:03] SMF on z-dat-s4-a is OK: OK - all services online [19:58:03] SMF on z-dat-s7-a is OK: OK - all services online [19:58:04] SMF on hyacinth is OK: OK - all services online [19:58:04] SMF on z-dat-s3-a is OK: OK - all services online [19:58:05] / on hyacinth is OK: DISK OK - free space: / 8430 MB (28% inode=85%): [19:58:05] SMF on z-dat-s6-a is OK: OK - all services online [19:58:10] Load avg. on hyacinth is OK: OK - load average: 5.48, 7.35, 7.78 [19:58:10] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:58:11] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:58:20] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:58:20] Load avg. on z-dat-s3-a is OK: OK - load average: 6.02, 7.42, 7.80 [19:58:30] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [20:03:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.431152/1.95, alarm hl:np_load_avg=1.417969/2.0, alarm hl:mem_free=278.000000M/350M, alarm hl:available=1/0 [20:05:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:10:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:11:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:12:01] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37134 MB (9% inode=99%): [20:15:11] ]@replag [20:15:18] @replag [20:15:19] matthewrbowker: s3-rr-a: 22s [-0.02 s/s]; s3-user: 22s [-0.02 s/s] [20:16:00] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22962 MB (5% inode=99%): [20:22:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.102539/1.95, alarm hl:np_load_avg=1.163574/2.0, alarm hl:mem_free=342.000000M/350M, alarm hl:available=1/0 [20:24:01] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 109857 MB (27% inode=99%): [20:24:50] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 389374 MB (7% inode=39%): [20:25:27] just to check I'm not being an idiot, editing crontab is "crontab -e", yes? [20:31:24] yes [20:32:20] hmm... something else is wrong then - the only output from that command is "The crontab file was not changed." - my editor doesn't get launched [20:33:02] what does "set | grep EDITOR" tell you? [20:33:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:33:17] bash $ set | grep EDITOR [20:33:17] EDITOR=/opt/ts/bin/nano [20:33:32] hm, that exists [20:33:34] weird... [20:34:07] works for me [20:34:33] you can also edit the file first [20:34:38] and then just do "crontab file" [20:34:48] ah, ok - I might try that then [20:35:02] hm, what's the deal with cronie and crontab on TS again? Let me check the wiki [20:36:04] ah, TS doc says use cronie, not crontab [20:36:07] https://wiki.toolserver.org/view/Crontab [20:36:43] I also like to have my crontab in a file and just read it in, since that means I always have a backup somewhere, and reinstating it is easy [20:37:06] that's not a bad idea [20:37:35] cronie seems to have the nonexistant /usr/bin/nano hardcoded into it :( [20:38:22] ouch [20:38:33] /bin/sh: /usr/bin/nano: not found // cronie: "/usr/bin/nano" exited with status 1 [20:39:24] wait, no, I'm an idiot [20:39:53] aparently it's the VISUAL environment var that was set wrong [20:40:28] oh, I see [20:40:42] I didn't realise I'd defined that in my .bashrc [20:40:44] nice catch [20:41:20] ty for the help :) [20:42:51] you're welcome, to the extent that I helped, happy to hear it got solved :) [20:51:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:52:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:55:01] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39471 MB (9% inode=99%): [21:01:40] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:06:01] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 103418 MB (25% inode=99%): [21:09:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.395508/1.95, alarm hl:np_load_avg=1.398926/2.0, alarm hl:mem_free=238.000000M/350M, alarm hl:available=1/0 [21:10:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:11:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:15:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:17:52] 3(resolved) [ACC-223] Impracticality of interface design <10https://jira.toolserver.org/browse/ACC-223> (Simon Walker) [21:17:53] 3(resolved) [ACC-244] add deleted contribs and hersfold to conflicting accounts <10https://jira.toolserver.org/browse/ACC-244> (Simon Walker) [21:21:53] 3(resolved) [ACC-192] Triage should check editcount of existing users <10https://jira.toolserver.org/browse/ACC-192> (Simon Walker) [21:24:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 389265 MB (7% inode=39%): [21:33:52] 3(closed) [ACC-225] IP address leak on internal interface <10https://jira.toolserver.org/browse/ACC-225> (Simon Walker) [21:34:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:35:53] 3(updated) [ACC-53] Use PHP's PDO library rather than using mysql_escape_string() <10https://jira.toolserver.org/browse/ACC-53> (Simon Walker) [21:35:57] 3(updated) [ACC-53] Use PHP's PDO library rather than using mysql_escape_string() <10https://jira.toolserver.org/browse/ACC-53> (Simon Walker) [21:37:52] 3(work started) [ACC-53] Use PHP's PDO library rather than using mysql_escape_string() <10https://jira.toolserver.org/browse/ACC-53> (Simon Walker) [21:39:54] 3(work logged) [ACC-53] Use PHP's PDO library rather than using mysql_escape_string() <10https://jira.toolserver.org/browse/ACC-53> (Simon Walker) [21:41:32] /sql on cassia is WARNING: DISK WARNING - free space: /sql 124668 MB (10% inode=99%): [21:49:56] 3(updated) [ACC-228] Squid IP shown as proxy, should be whitelisted <10https://jira.toolserver.org/browse/ACC-228> (Simon Walker) [21:51:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:53:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.991699/1.95, alarm hl:np_load_avg=1.062012/2.0, alarm hl:mem_free=147.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.991699/2.3, alarm hl:np_load_long=1.036621/2.5, alarm hl:cpu=74.600000/98, alarm hl:mem_free=147.000000M/150M, alarm hl:available=1/0 [21:57:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:07:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.410645/1.95, alarm hl:np_load_avg=1.436035/2.0, alarm hl:mem_free=294.000000M/350M, alarm hl:available=1/0 [22:10:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:11:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:13:11] Load avg. on willow is WARNING: WARNING - load average: 14.11, 15.11, 12.30 [22:13:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.700195/1.10, alarm hl:np_load_long=0.690430/1.55, alarm hl:mem_free=17937.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.700195/1.00, alarm hl:np_load_long=0.690430/1.50, alarm hl:mem_free=17937.000000M/600M, alarm hl:available=1/0 [22:14:11] Load avg. on willow is OK: OK - load average: 11.86, 14.42, 12.25 [22:14:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:15:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:19:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.422851/1.10, alarm hl:np_load_long=0.841797/1.55, alarm hl:mem_free=17696.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.422851/1.00, alarm hl:np_load_long=0.841797/1.50, alarm hl:mem_free=17696.000000M/600M, alarm hl:available=1/0 [22:24:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 389204 MB (7% inode=39%): [22:27:05] [[Special:Log/delete]] delete 10 * Ltcmdrap * (deleted "[[02User:Tparis/Index10]]": Non existant user) [22:34:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:38:53] [[Special:Log/newusers]] create 10 * Lpdreklama * (New user account) [22:42:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.005859/1.95, alarm hl:np_load_avg=1.073242/2.0, alarm hl:mem_free=205.000000M/350M, alarm hl:available=1/0 [22:46:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:50:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.623047/1.95, alarm hl:np_load_avg=1.247559/2.0, alarm hl:mem_free=193.000000M/350M, alarm hl:available=1/0 [22:51:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:11:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:11:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 389141 MB (7% inode=39%): [23:28:31] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1936.000000 [23:31:55] 3(work logged) [ACC-53] Use PHP's PDO library rather than using mysql_escape_string() <10https://jira.toolserver.org/browse/ACC-53> (Simon Walker) [23:34:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:51:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk