[00:08:41] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8355.000000 [00:09:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:13:12] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:14:44] 3(closed) [PATHOSCHILD-10] Formatting of TemplateScript menus is inverted <10https://jira.toolserver.org/browse/PATHOSCHILD-10> (Jesse Plamondon-Willard) [00:19:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:20:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:26:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:26:21] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:32:31] [[User:Ellajqp8713]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7314&rcid=9794 * Ellajqp8713 * (+207) (Created page with "What's up? My name's Toni Singh, I have been impressed with how much is up on this project, and I'm looking forward to contributing, too. My page: [http://www.wedding-cake-stan...") [00:33:37] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Ellajqp871310]] with an expiry time of infinite (account creation disabled): Spam) [00:33:45] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02User:Ellajqp871310]]": Spam) [00:46:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:50:41] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 184161 MB (3% inode=23%): [01:08:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9058.000000 [01:09:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:14:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:20:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:20:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:26:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:28:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.765137/1.95, alarm hl:tmp_free=38852M/100M, alarm hl:np_load_avg=0.846191/2.0, alarm hl:mem_free=232.000000M/350M, alarm hl:available=1/0 [01:31:20] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [01:48:32] [[Special:Log/newusers]] create 10 * Maragaretdago152 * (New user account) [01:51:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 216238 MB (4% inode=26%): [02:01:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.467285/1.95, alarm hl:tmp_free=38657M/100M, alarm hl:np_load_avg=1.285645/2.0, alarm hl:mem_free=122.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.467285/2.3, alarm hl:np_load_long=1.193359/2.5, alarm hl:cpu=84.100000/98, alarm hl:mem_free=122.000000M/200M, al [02:08:21] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:09:01] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5876.000000 [02:09:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:12:41] Load avg. on willow is WARNING: WARNING - load average: 20.04, 16.05, 12.73 [02:13:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.446289/1.95, alarm hl:tmp_free=38599M/100M, alarm hl:np_load_avg=2.039062/2.0, alarm hl:mem_free=156.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.446289/2.3, alarm hl:np_load_long=1.617676/2.5, alarm hl:cpu=99.500000/98, alarm hl:mem_free=156.000000M/200M, al [02:14:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:17:21] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:17:41] Load avg. on willow is OK: OK - load average: 10.20, 14.41, 13.12 [02:20:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:20:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:24:01] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3547.000000 [02:26:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:32:41] Load avg. on willow is WARNING: WARNING - load average: 16.52, 14.40, 12.83 [02:39:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.086914/1.95, alarm hl:tmp_free=38474M/100M, alarm hl:np_load_avg=1.309082/2.0, alarm hl:mem_free=190.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.086914/2.3, alarm hl:np_load_long=1.442871/2.5, alarm hl:cpu=73.700000/98, alarm hl:mem_free=190.000000M/200M, al [02:44:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:49:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.208984/1.95, alarm hl:tmp_free=38427M/100M, alarm hl:np_load_avg=1.332520/2.0, alarm hl:mem_free=191.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.208984/2.3, alarm hl:np_load_long=1.418945/2.5, alarm hl:cpu=92.100000/98, alarm hl:mem_free=191.000000M/200M, al [02:50:48] [[User:Betacommand/common.js]] N 10https://wiki.toolserver.org/w/index.php?oldid=7315&rcid=9798 * Betacommand * (+603) (Created page with "importScript('User:Lupin/popups.js', 'en'); importScriptURI('//en.wikipedia.org/w/index.php?title=User:Lupin/popups.js&action=raw&ctype=text/javascript'); //can I just use the ...") [02:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 208903 MB (3% inode=25%): [02:51:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:53:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1690.000000 [02:59:09] [[MediaWiki:Titleblacklist]] 10https://wiki.toolserver.org/w/index.php?diff=7316&oldid=6448&rcid=9799 * Betacommand * (+18) (Lets see if I can limit page creation to new accounts only to stop this spam) [03:00:39] [[Special:Log/newusers]] create 10 * Betacommand TEST * (New user account) [03:00:56] [[E]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7317&rcid=9801 * Betacommand TEST * (+4) (Created page with "test") [03:01:25] [[MediaWiki:Titleblacklist]] 10https://wiki.toolserver.org/w/index.php?diff=7318&oldid=7316&rcid=9802 * Betacommand * (+0) () [03:01:40] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02E10]]": content was: "test" (and the only contributor was "[[Special:Contributions/Betacommand TEST|Betacommand TEST]]")) [03:01:57] [[E]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7319&rcid=9804 * Betacommand TEST * (+1) (Created page with "e") [03:05:06] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02E10]]": content was: "e" (and the only contributor was "[[Special:Contributions/Betacommand TEST|Betacommand TEST]]")) [03:05:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.918945/1.95, alarm hl:tmp_free=38342M/100M, alarm hl:np_load_avg=1.680664/2.0, alarm hl:mem_free=270.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.918945/2.3, alarm hl:np_load_long=1.497559/2.5, alarm hl:cpu=98.600000/98, alarm hl:mem_free=270.000000M/200M, al [03:07:44] Load avg. on willow is WARNING: WARNING - load average: 19.02, 15.36, 12.92 [03:08:45] 3(created) [TS-1392] Install Extension:TitleBlacklist; Toolserver: Wiki; Task <10https://jira.toolserver.org/browse/TS-1392> (Betacommand) [03:09:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:09:44] Load avg. on willow is OK: OK - load average: 14.10, 14.85, 13.04 [03:12:44] Load avg. on willow is WARNING: WARNING - load average: 17.98, 16.71, 14.13 [03:14:23] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:18:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:20:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:20:24] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.136719/1.10, alarm hl:np_load_long=0.761719/1.55, alarm hl:mem_free=10305.000000M/500M, alarm hl:tmp_free=15535M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.136719/1.00, alarm hl:np_load_long=0.761719/1.50, alarm hl:mem_free=10305.000000M/600M, alarm hl:tmp_free [03:20:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:22:24] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:26:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:28:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.095215/1.95, alarm hl:tmp_free=38234M/100M, alarm hl:np_load_avg=1.281250/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0 [03:31:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:51:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 202800 MB (3% inode=25%): [04:02:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.501953/1.95, alarm hl:tmp_free=38023M/100M, alarm hl:np_load_avg=1.294922/2.0, alarm hl:mem_free=218.000000M/350M, alarm hl:available=1/0 [04:07:44] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:09:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:10:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.610840/1.95, alarm hl:tmp_free=37971M/100M, alarm hl:np_load_avg=1.923340/2.0, alarm hl:mem_free=329.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.610840/2.3, alarm hl:np_load_long=1.509277/2.5, alarm hl:cpu=99.600000/98, alarm hl:mem_free=329.000000M/200M, al [04:12:54] Load avg. on willow is WARNING: WARNING - load average: 19.70, 16.90, 13.14 [04:14:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:18:55] Load avg. on willow is OK: OK - load average: 9.75, 14.47, 13.58 [04:20:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:20:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:26:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:51:22] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199289 MB (3% inode=24%): [04:57:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.494629/1.95, alarm hl:tmp_free=37593M/100M, alarm hl:np_load_avg=1.358399/2.0, alarm hl:mem_free=174.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.494629/2.3, alarm hl:np_load_long=1.412598/2.5, alarm hl:cpu=95.300000/98, alarm hl:mem_free=174.000000M/200M, al [04:59:52] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:10:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:14:52] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:21:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:21:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:26:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:26:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:31:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.821289/1.95, alarm hl:tmp_free=37424M/100M, alarm hl:np_load_avg=1.421387/2.0, alarm hl:mem_free=337.000000M/350M, alarm hl:available=1/0 [05:32:02] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:35:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.072754/1.95, alarm hl:tmp_free=37389M/100M, alarm hl:np_load_avg=1.202149/2.0, alarm hl:mem_free=251.000000M/350M, alarm hl:available=1/0 [05:41:22] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:51:22] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 197055 MB (3% inode=24%): [05:55:23] /sql on cassia is WARNING: DISK WARNING - free space: /sql 117949 MB (9% inode=99%): [06:09:22] Load avg. on willow is WARNING: WARNING - load average: 18.20, 15.09, 12.92 [06:10:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:10:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.359375/1.95, alarm hl:tmp_free=36916M/100M, alarm hl:np_load_avg=1.944824/2.0, alarm hl:mem_free=581.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.359375/2.3, alarm hl:np_load_long=1.643555/2.5, alarm hl:cpu=99.800000/98, alarm hl:mem_free=581.000000M/200M, al [06:15:02] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:17:02] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:17:22] Load avg. on willow is OK: OK - load average: 9.25, 14.29, 14.06 [06:21:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:21:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:27:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:51:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 195571 MB (3% inode=24%): [07:08:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.071777/1.95, alarm hl:tmp_free=36616M/100M, alarm hl:np_load_avg=1.719238/2.0, alarm hl:mem_free=410.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.071777/2.3, alarm hl:np_load_long=1.520508/2.5, alarm hl:cpu=99.700000/98, alarm hl:mem_free=410.000000M/200M, al [07:08:21] Load avg. on willow is WARNING: WARNING - load average: 17.00, 14.05, 12.30 [07:10:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:15:02] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:15:02] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [07:17:21] Load avg. on willow is OK: OK - load average: 10.85, 14.14, 13.55 [07:21:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:21:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:27:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:40:42] Load avg. on adenia is WARNING: WARNING - load average: 18.77, 13.79, 9.11 [07:44:42] Load avg. on adenia is OK: OK - load average: 14.89, 14.59, 10.55 [07:51:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 195455 MB (3% inode=24%): [07:55:42] 3(created) [DBQ-185] wanted pages in fa.wikipediad; Database Queries; Task <10https://jira.toolserver.org/browse/DBQ-185> (reza) [08:09:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.562012/1.95, alarm hl:tmp_free=36339M/100M, alarm hl:np_load_avg=1.502441/2.0, alarm hl:mem_free=209.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.562012/2.3, alarm hl:np_load_long=1.465820/2.5, alarm hl:cpu=98.300000/98, alarm hl:mem_free=209.000000M/200M, al [08:10:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:12:42] Load avg. on willow is WARNING: WARNING - load average: 15.21, 13.98, 12.62 [08:15:12] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:15:12] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:16:42] Load avg. on willow is OK: OK - load average: 10.97, 13.82, 13.00 [08:21:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:21:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:28:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:38:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.076172/1.95, alarm hl:tmp_free=36209M/100M, alarm hl:np_load_avg=1.157715/2.0, alarm hl:mem_free=238.000000M/350M, alarm hl:available=1/0 [08:39:12] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:43:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.086426/1.95, alarm hl:tmp_free=36185M/100M, alarm hl:np_load_avg=1.150391/2.0, alarm hl:mem_free=245.000000M/350M, alarm hl:available=1/0 [08:51:01] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:51:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 196084 MB (3% inode=24%): [08:52:02] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [08:56:42] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:01:32] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:10:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:12:53] Load avg. on willow is WARNING: WARNING - load average: 16.64, 15.06, 12.23 [09:15:22] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:16:52] Load avg. on willow is OK: OK - load average: 11.08, 14.08, 12.58 [09:18:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42123 MB (10% inode=99%): [09:21:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:21:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:28:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:28:53] Load avg. on willow is WARNING: WARNING - load average: 22.89, 18.79, 15.23 [09:35:42] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63379 MB (15% inode=99%): [09:38:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.438477/1.95, alarm hl:tmp_free=35932M/100M, alarm hl:np_load_avg=2.686523/2.0, alarm hl:mem_free=216.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.438477/2.3, alarm hl:np_load_long=2.256348/2.5, alarm hl:cpu=99.800000/98, alarm hl:mem_free=216.000000M/200M, al [09:47:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:52:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 193834 MB (3% inode=24%): [09:52:52] Load avg. on willow is OK: OK - load average: 9.49, 11.83, 14.78 [09:59:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.691895/1.95, alarm hl:tmp_free=35843M/100M, alarm hl:np_load_avg=1.608887/2.0, alarm hl:mem_free=346.000000M/350M, alarm hl:available=1/0 [10:01:52] Load avg. on willow is WARNING: WARNING - load average: 26.19, 19.93, 16.78 [10:10:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:15:22] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:17:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [10:18:01] Free Memory on damiana is WARNING: WARNING - 6.6% (275260 kB) free! [10:19:52] Load avg. on willow is OK: OK - load average: 6.92, 11.47, 14.86 [10:21:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:22:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:28:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:29:23] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:29:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.336426/1.95, alarm hl:tmp_free=35672M/100M, alarm hl:np_load_avg=1.317871/2.0, alarm hl:mem_free=299.000000M/350M, alarm hl:available=1/0 [10:29:52] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:30:03] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:30:32] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:31:04] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:31:12] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 222.000000 [10:31:12] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:31:12] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:31:22] MySQL slave on z-dat-s3-a is OK: Uptime: 2643033 Threads: 24 Questions: 2851732335 Slow queries: 177587 Opens: 27752304 Flush tables: 1 Open tables: 16384 Queries per second avg: 1078.962 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 254 [10:31:23] SMF on z-dat-s6-a is OK: OK - all services online [10:31:32] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:31:32] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:31:42] MySQL slave on z-dat-s6-a is OK: Uptime: 702553 Threads: 13 Questions: 147353107 Slow queries: 45798 Opens: 1196888 Flush tables: 1 Open tables: 2429 Queries per second avg: 209.739 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 236 [10:31:55] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [10:32:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [10:49:47] 3(updated) [ACCAPP-506] ArticleNet: Development of a grafical tool to show relations between WP articles <10https://jira.toolserver.org/browse/ACCAPP-506> (Claus Colloseus) [10:52:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 190835 MB (3% inode=24%): [11:08:02] Load avg. on willow is WARNING: WARNING - load average: 16.84, 14.68, 12.81 [11:10:02] Load avg. on willow is OK: OK - load average: 13.77, 14.63, 13.05 [11:10:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:15:32] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:18:02] Free Memory on damiana is WARNING: WARNING - 5.8% (241844 kB) free! [11:21:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:22:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:28:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:52:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 188923 MB (3% inode=23%): [12:00:22] Free Memory on damiana is OK: OK - 7.1% (296376 kB) free. [12:01:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.297852/1.95, alarm hl:tmp_free=35096M/100M, alarm hl:np_load_avg=1.464844/2.0, alarm hl:mem_free=667.000000M/350M, alarm hl:available=1/0 [12:02:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [12:03:23] Free Memory on damiana is WARNING: WARNING - 6.3% (263568 kB) free! [12:07:22] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (184192 kB) free! [12:07:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.607422/1.95, alarm hl:tmp_free=35035M/100M, alarm hl:np_load_avg=1.501465/2.0, alarm hl:mem_free=191.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.607422/2.3, alarm hl:np_load_long=1.356934/2.5, alarm hl:cpu=98.600000/98, alarm hl:mem_free=191.000000M/200M, al [12:10:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:15:42] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:21:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:22:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:22:43] Quick question; does anyone have a recent copy of enwiki's LocalSettings.php ? Sepcifically interested in the values used on live for $wgDefaultRobotPolicy, $wgNamespaceRobotPolicies and $wgArticleRobotPolicies [12:26:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.723633/1.10, alarm hl:np_load_long=1.237305/1.55, alarm hl:mem_free=9187.000000M/500M, alarm hl:tmp_free=15073M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.723633/1.00, alarm hl:np_load_long=1.237305/1.50, alarm hl:mem_free=9187.000000M/600M, alarm hl:tmp_free=1 [12:27:43] Load avg. on ortelius is WARNING: WARNING - load average: 15.11, 10.61, 5.70 [12:28:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:28:43] Load avg. on ortelius is OK: OK - load average: 13.25, 10.83, 6.08 [12:31:42] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:34:22] Free Memory on damiana is WARNING: WARNING - 6.3% (262464 kB) free! [12:39:27] Free Memory on damiana is OK: OK - 7.1% (298004 kB) free. [12:46:47] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:50:27] Free Memory on damiana is WARNING: WARNING - 6.5% (273488 kB) free! [12:51:27] Free Memory on damiana is OK: OK - 7.2% (300828 kB) free. [12:52:27] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 189317 MB (3% inode=23%): [12:54:43] 3(created) [TS-1393] Fix TclCurl pkgIndex.tcl; Toolserver: Software installation; Bug <10https://jira.toolserver.org/browse/TS-1393> (gifti) [13:10:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:11:26] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:15:47] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:21:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:22:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:26:38] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1903.000000 [13:28:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:33:58] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.131836/1.95, alarm hl:tmp_free=34616M/100M, alarm hl:np_load_avg=1.144531/2.0, alarm hl:mem_free=264.000000M/350M, alarm hl:available=1/0 [13:34:57] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [13:38:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.071289/1.95, alarm hl:tmp_free=34591M/100M, alarm hl:np_load_avg=1.071777/2.0, alarm hl:mem_free=324.000000M/350M, alarm hl:available=1/0 [13:52:38] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 187535 MB (3% inode=23%): [13:58:43] 3(created) [PATHOSCHILD-11] TemplateScript menus should "remember" if they were expanded or collapsed; Pathoschild's tools; Bug <10https://jira.toolserver.org/browse/PATHOSCHILD-11> (Helder Wiki) [14:06:40] Load avg. on willow is WARNING: WARNING - load average: 20.77, 15.71, 11.81 [14:07:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.540039/1.95, alarm hl:tmp_free=34461M/100M, alarm hl:np_load_avg=1.965820/2.0, alarm hl:mem_free=405.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.540039/2.3, alarm hl:np_load_long=1.480469/2.5, alarm hl:cpu=97.800000/98, alarm hl:mem_free=405.000000M/200M, al [14:08:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:08:37] Load avg. on willow is OK: OK - load average: 11.59, 14.22, 11.75 [14:11:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:14:07] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.804199/1.95, alarm hl:tmp_free=34429M/100M, alarm hl:np_load_avg=1.795899/2.0, alarm hl:mem_free=206.000000M/350M, alarm hl:available=1/0 [14:16:07] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:22:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:23:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:26:37] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2627.000000 [14:29:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:51:37] Load avg. on willow is WARNING: WARNING - load average: 20.25, 15.58, 12.64 [14:52:37] Load avg. on willow is OK: OK - load average: 14.05, 14.66, 12.50 [14:53:38] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 183588 MB (3% inode=23%): [14:54:37] Free Memory on damiana is WARNING: WARNING - 7.0% (294480 kB) free! [14:56:38] Free Memory on damiana is OK: OK - 7.2% (300344 kB) free. [15:04:37] Free Memory on damiana is WARNING: WARNING - 7.0% (292100 kB) free! [15:11:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:16:17] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:22:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:23:37] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:26:47] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3304.000000 [15:29:17] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:29:48] Free Memory on damiana is WARNING: WARNING - 6.4% (266760 kB) free! [15:45:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3616.000000 [15:53:47] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181224 MB (3% inode=23%): [16:11:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:16:27] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:22:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:23:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:29:17] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:29:56] Free Memory on damiana is WARNING: WARNING - 6.8% (285880 kB) free! [16:32:18] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.223145/1.95, alarm hl:tmp_free=33837M/100M, alarm hl:np_load_avg=1.095703/2.0, alarm hl:mem_free=318.000000M/350M, alarm hl:available=1/0 [16:33:17] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:38:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.992188/1.95, alarm hl:tmp_free=33809M/100M, alarm hl:np_load_avg=0.971680/2.0, alarm hl:mem_free=290.000000M/350M, alarm hl:available=1/0 [16:45:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6207.000000 [16:54:47] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181241 MB (3% inode=23%): [17:11:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:16:26] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:22:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:23:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:29:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:30:07] Free Memory on damiana is WARNING: WARNING - 6.7% (278712 kB) free! [17:45:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7845.000000 [17:54:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181134 MB (3% inode=23%): [17:55:27] /sql on cassia is WARNING: DISK WARNING - free space: /sql 116168 MB (9% inode=99%): [18:11:37] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:12:57] Load avg. on willow is WARNING: WARNING - load average: 17.55, 15.45, 12.29 [18:14:57] Load avg. on willow is OK: OK - load average: 12.38, 14.44, 12.32 [18:16:37] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:22:37] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:24:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:29:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:30:07] Free Memory on damiana is WARNING: WARNING - 5.6% (236196 kB) free! [18:34:05] DaBPunkt: when i write a script to check for dead external links are there any limits for http requests that i have to obey? [18:40:58] common sense [18:42:07] mh [18:45:28] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8464.000000 [18:54:58] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 181038 MB (3% inode=23%): [19:11:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:16:47] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:22:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:24:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:27:17] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (207992 kB) free! [19:28:19] Free Memory on damiana is WARNING: WARNING - 6.3% (265388 kB) free! [19:28:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.233887/1.95, alarm hl:tmp_free=32747M/100M, alarm hl:np_load_avg=1.218262/2.0, alarm hl:mem_free=137.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.233887/2.3, alarm hl:np_load_long=1.253906/2.5, alarm hl:cpu=84.500000/98, alarm hl:mem_free=137.000000M/200M, al [19:29:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:30:48] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:35:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.091309/1.95, alarm hl:tmp_free=32690M/100M, alarm hl:np_load_avg=1.142090/2.0, alarm hl:mem_free=201.000000M/350M, alarm hl:available=1/0 [19:45:28] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10211.000000 [19:50:48] thekla: Do you know of Checklinks? [19:54:16] yeah [19:55:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 180917 MB (3% inode=23%): [20:01:28] and it is so slow. it took 7 minutes to detect/check all links [20:07:55] oh, heh, that was me :( [20:09:30] thekla: It's rate limited to prevent abuse [20:09:39] hm, ok [20:09:52] 2 sec delay per link, 15+ threads [20:14:16] ooops, now that damiana seems down, everything seems broken, ssh, webservice [20:15:12] The data is also public to other toolserver user. No time yet to put into a nice format though [20:15:34] where is the data located? [20:16:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:16:51] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:17:17] /home/dispenser/public_html/cgi-bin/cache/ It fucking huge, 2+ GB like 100,000 pages [20:17:29] phew [20:18:45] There's a script that retouches the cache, https://en.wikipedia.org/wiki/User:Dispenser/Checklinks/config [20:19:50] A nice tool to shame webmaster is "broken links by domain" [20:21:44] 3(updated) [ACCAPP-508] I would like to have Toolserver access for running SQL queries against enwiki <10https://jira.toolserver.org/browse/ACCAPP-508> (Phuzion) [20:23:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:24:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:30:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:30:44] 3(resolved) [DBQ-185] wanted pages in fa.wikipediad <10https://jira.toolserver.org/browse/DBQ-185> (Hoo man) [20:32:44] 3(commented) [ACCAPP-508] I would like to have Toolserver access for running SQL queries against enwiki <10https://jira.toolserver.org/browse/ACCAPP-508> (Dispenser) [20:36:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:40:44] 3(commented) [DBQ-185] wanted pages in fa.wikipediad <10https://jira.toolserver.org/browse/DBQ-185> (reza) [20:46:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12908.000000 [20:49:13] @replag [20:49:14] DaBPunkt: s1-user: 6m 39s [-]; s2-user: 2d 14h 53s [-]; s2-user-c: 3h 38m 10s [-]; s5-user-c: 3h 38m 11s [-] [20:55:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 180597 MB (3% inode=23%): [21:13:11] Load avg. on willow is WARNING: WARNING - load average: 16.04, 13.28, 10.86 [21:14:11] Load avg. on willow is OK: OK - load average: 13.52, 13.20, 10.99 [21:16:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:17:01] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:23:12] heh. qsub is not happy when damiana is unresponsive. [21:23:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:24:11] Load avg. on willow is WARNING: WARNING - load average: 20.12, 15.39, 12.55 [21:25:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:30:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:34:44] 3(commented) [DBQ-185] wanted pages in fa.wikipediad <10https://jira.toolserver.org/browse/DBQ-185> (Hoo man) [21:36:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:46:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7519.000000 [21:48:44] 3(commented) [ACCAPP-508] I would like to have Toolserver access for running SQL queries against enwiki <10https://jira.toolserver.org/browse/ACCAPP-508> (Phuzion) [21:56:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 179902 MB (3% inode=23%): [22:05:50] nacht ts [22:12:21] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3551.000000 [22:13:11] Load avg. on willow is WARNING: WARNING - load average: 16.63, 14.74, 12.99 [22:15:10] Load avg. on willow is OK: OK - load average: 11.96, 13.97, 12.94 [22:17:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:17:02] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:22:21] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1725.000000 [22:24:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:25:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:30:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:37:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:57:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 179899 MB (3% inode=23%): [23:17:11] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:18:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:26:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:30:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:37:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:58:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 178523 MB (3% inode=22%):