[00:05:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:06:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:08:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.616211/1.95, alarm hl:tmp_free=29044M/100M, alarm hl:np_load_avg=0.626465/2.0, alarm hl:mem_free=239.000000M/350M, alarm hl:available=1/0 [00:10:04] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:14:13] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:14:14] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:16:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:23:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:32:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.849609/1.95, alarm hl:tmp_free=28925M/100M, alarm hl:np_load_avg=0.683594/2.0, alarm hl:mem_free=244.000000M/350M, alarm hl:available=1/0 [00:33:14] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:47:44] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 195270 MB (3% inode=24%): [00:50:44] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6357.000000 [01:00:54] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:05:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:10:04] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:14:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:16:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:23:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:26:43] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3541.000000 [01:37:54] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1767.000000 [01:38:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.435547/1.95, alarm hl:tmp_free=28592M/100M, alarm hl:np_load_avg=0.526367/2.0, alarm hl:mem_free=296.000000M/350M, alarm hl:available=1/0 [01:39:13] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [01:47:44] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 206846 MB (3% inode=25%): [02:05:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:10:06] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:10:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.747070/1.95, alarm hl:tmp_free=28422M/100M, alarm hl:np_load_avg=0.785156/2.0, alarm hl:mem_free=260.000000M/350M, alarm hl:available=1/0 [02:14:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:15:17] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:16:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:23:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:23:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.639160/1.95, alarm hl:tmp_free=28356M/100M, alarm hl:np_load_avg=0.814453/2.0, alarm hl:mem_free=331.000000M/350M, alarm hl:available=1/0 [02:43:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:48:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201065 MB (3% inode=25%): [03:01:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.450684/1.95, alarm hl:tmp_free=28149M/100M, alarm hl:np_load_avg=1.307129/2.0, alarm hl:mem_free=597.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.450684/2.3, alarm hl:np_load_long=0.952637/2.5, alarm hl:cpu=80.000000/98, alarm hl:mem_free=597.000000M/200M, al [03:02:29] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:05:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:10:08] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:12:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.063965/1.95, alarm hl:tmp_free=28084M/100M, alarm hl:np_load_avg=1.033691/2.0, alarm hl:mem_free=189.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.063965/2.3, alarm hl:np_load_long=0.961914/2.5, alarm hl:cpu=81.600000/98, alarm hl:mem_free=189.000000M/200M, al [03:14:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:16:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:23:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:48:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 200655 MB (3% inode=25%): [04:05:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:10:08] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:15:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:16:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:18:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.823242/1.95, alarm hl:tmp_free=27729M/100M, alarm hl:np_load_avg=1.155274/2.0, alarm hl:mem_free=188.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.823242/2.3, alarm hl:np_load_long=1.151855/2.5, alarm hl:cpu=70.000000/98, alarm hl:mem_free=188.000000M/200M, al [04:23:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:46:30] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:48:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 195190 MB (3% inode=24%): [04:54:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.662598/1.95, alarm hl:tmp_free=27545M/100M, alarm hl:np_load_avg=0.644531/2.0, alarm hl:mem_free=183.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.662598/2.3, alarm hl:np_load_long=0.724609/2.5, alarm hl:cpu=66.700000/98, alarm hl:mem_free=183.000000M/200M, al [04:57:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:05:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:10:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:11:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.507324/1.95, alarm hl:tmp_free=27433M/100M, alarm hl:np_load_avg=1.104980/2.0, alarm hl:mem_free=200.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.507324/2.3, alarm hl:np_load_long=0.938477/2.5, alarm hl:cpu=93.600000/98, alarm hl:mem_free=200.000000M/200M, al [05:13:58] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:14:58] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [05:15:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:16:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:16:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:21:18] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:23:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:30:58] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:47:42] 3(resolved) [PATHOSCHILD-6] Add regex tool to TemplateScript <10https://jira.toolserver.org/browse/PATHOSCHILD-6> (Jesse Plamondon-Willard) [05:48:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 216659 MB (4% inode=26%): [05:49:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.163574/1.95, alarm hl:tmp_free=27219M/100M, alarm hl:np_load_avg=1.231934/2.0, alarm hl:mem_free=196.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.163574/2.3, alarm hl:np_load_long=1.114258/2.5, alarm hl:cpu=61.700000/98, alarm hl:mem_free=196.000000M/200M, al [05:54:17] /sql on cassia is WARNING: DISK WARNING - free space: /sql 117345 MB (9% inode=99%): [05:54:42] 3(closed) [PATHOSCHILD-6] Add regex tool to TemplateScript <10https://jira.toolserver.org/browse/PATHOSCHILD-6> (Jesse Plamondon-Willard) [06:00:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:03:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.132324/1.95, alarm hl:tmp_free=27049M/100M, alarm hl:np_load_avg=1.526367/2.0, alarm hl:mem_free=179.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.132324/2.3, alarm hl:np_load_long=1.255371/2.5, alarm hl:cpu=98.900000/98, alarm hl:mem_free=179.000000M/200M, al [06:05:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:10:18] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:12:09] Load avg. on willow is WARNING: WARNING - load average: 15.56, 15.38, 12.45 [06:13:18] Load avg. on willow is OK: OK - load average: 11.05, 14.16, 12.21 [06:15:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:16:27] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:16:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:23:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:38:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.761231/1.95, alarm hl:tmp_free=26824M/100M, alarm hl:np_load_avg=0.965332/2.0, alarm hl:mem_free=310.000000M/350M, alarm hl:available=1/0 [06:48:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 214350 MB (4% inode=26%): [06:50:38] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:55:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.699219/1.95, alarm hl:tmp_free=26741M/100M, alarm hl:np_load_avg=0.889648/2.0, alarm hl:mem_free=232.000000M/350M, alarm hl:available=1/0 [07:00:39] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [07:05:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:10:28] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:15:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:16:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:23:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:36:28] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:36:37] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.871582/1.95, alarm hl:tmp_free=26530M/100M, alarm hl:np_load_avg=0.938965/2.0, alarm hl:mem_free=186.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.871582/2.3, alarm hl:np_load_long=0.929199/2.5, alarm hl:cpu=64.800000/98, alarm hl:mem_free=186.000000M/200M, al [07:38:38] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [07:40:59] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:42:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.677246/1.95, alarm hl:tmp_free=26499M/100M, alarm hl:np_load_avg=0.827148/2.0, alarm hl:mem_free=206.000000M/350M, alarm hl:available=1/0 [07:48:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 211742 MB (3% inode=26%): [07:48:39] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:05:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:10:27] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:15:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:16:28] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1828.000000 [08:17:37] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:23:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:48:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209642 MB (3% inode=25%): [08:55:07] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:08] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:29] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:29] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:29] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:30] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:30] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:30] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:30] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:39] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:55:58] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [08:55:58] Load avg. on z-dat-s6-a is OK: OK - load average: 1.36, 1.92, 2.34 [08:55:58] / on z-dat-s6-a is OK: DISK OK - free space: / 8283 MB (27% inode=85%): [08:55:59] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 140467 MB (14% inode=99%): [08:56:18] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:56:18] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:56:18] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:56:18] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:04:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [09:05:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:06:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.141601/1.95, alarm hl:tmp_free=26084M/100M, alarm hl:np_load_avg=1.124024/2.0, alarm hl:mem_free=281.000000M/350M, alarm hl:available=1/0 [09:08:38] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:10:37] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:15:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:17:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:22:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.750000/1.95, alarm hl:tmp_free=26016M/100M, alarm hl:np_load_avg=0.820801/2.0, alarm hl:mem_free=278.000000M/350M, alarm hl:available=1/0 [09:23:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:30:37] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:33:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.681641/1.95, alarm hl:tmp_free=25965M/100M, alarm hl:np_load_avg=0.816894/2.0, alarm hl:mem_free=249.000000M/350M, alarm hl:available=1/0 [09:48:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 206111 MB (3% inode=25%): [10:05:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4891.000000 [10:05:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:10:38] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:16:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:17:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:23:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:48:27] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 204159 MB (3% inode=25%): [10:55:44] 3(created) [TS-1391] Enable SSH functions in PHP (ssh2_connect, ssh2_exec); Toolserver; Task <10https://jira.toolserver.org/browse/TS-1391> (Krinkle) [11:04:39] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.399414/1.10, alarm hl:np_load_long=0.635742/1.55, alarm hl:mem_free=9849.000000M/500M, alarm hl:tmp_free=15059M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.399414/1.00, alarm hl:np_load_long=0.635742/1.50, alarm hl:mem_free=9849.000000M/600M, alarm hl:tmp_free=1 [11:05:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7119.000000 [11:05:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:07:38] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:10:48] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:16:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:17:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:23:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:35:46] [[Cronietab]] N 10https://wiki.toolserver.org/w/index.php?oldid=7301&rcid=9767 * Krinkle * (+17) (Redirected page to [[Cron]]) [11:39:53] [[Cronie]] 10https://wiki.toolserver.org/w/index.php?diff=7302&oldid=6723&rcid=9768 * Krinkle * (+7) (Redirected page to [[Cron#Cronie]]) [11:40:03] [[Cronietab]] 10https://wiki.toolserver.org/w/index.php?diff=7303&oldid=7301&rcid=9769 * Krinkle * (+10) (Redirected page to [[Cron#Cronietab]]) [11:41:08] [[Cron]] 10https://wiki.toolserver.org/w/index.php?diff=7304&oldid=5196&rcid=9770 * Krinkle * (+798) (+Cronietab documentation) [11:42:49] [[Submit]] N 10https://wiki.toolserver.org/w/index.php?oldid=7305&rcid=9771 * Krinkle * (+49) (Redirected page to [[Job scheduling#submit.toolserver.org]]) [11:42:58] [[Admin:Submit]] N 10https://wiki.toolserver.org/w/index.php?oldid=7306&rcid=9772 * Krinkle * (+49) (Redirected page to [[Job scheduling#submit.toolserver.org]]) [11:43:18] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02Admin:Submit10]]": Redundant page) [11:43:29] [[Submit.toolserver.org]] NM 10https://wiki.toolserver.org/w/index.php?oldid=7307&rcid=9774 * Krinkle * (+49) (Redirected page to [[Job scheduling#submit.toolserver.org]]) [11:43:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.147461/1.10, alarm hl:np_load_long=0.817383/1.55, alarm hl:mem_free=10055.000000M/500M, alarm hl:tmp_free=15092M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.147461/1.00, alarm hl:np_load_long=0.817383/1.50, alarm hl:mem_free=10055.000000M/600M, alarm hl:tmp_free [11:43:45] [[Job scheduling]] 10https://wiki.toolserver.org/w/index.php?diff=7308&oldid=7107&rcid=9775 * Krinkle * (+26) (/* submit.toolserver.org */ +link to cronietab docs) [11:44:12] [[Cron]] M 10https://wiki.toolserver.org/w/index.php?diff=7309&oldid=7304&rcid=9776 * Krinkle * (+1) () [11:45:10] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02Avaya 3000 exam10]]": Spam) [11:45:42] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:46:07] [[MediaWiki:Ipbreason-dropdown]] NM 10https://wiki.toolserver.org/w/index.php?oldid=7310&rcid=9778 * Krinkle * (+203) (Simplify) [11:46:49] [[Special:Log/block]] block 10 * Krinkle * (blocked [[02User:Avaya300010]] with an expiry time of infinite (account creation disabled, e-mail blocked): Spam) [11:47:10] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02VCP510 exam10]]": Spam) [11:47:21] [[Special:Log/block]] block 10 * Krinkle * (blocked [[02User:VCP510-exam10]] with an expiry time of infinite (account creation disabled, e-mail blocked): Spam) [11:47:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41999 MB (10% inode=99%): [11:47:28] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7281 of [[02Wiki server assignments10]] patrolled ) [11:47:46] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7276 of [[02Bots10]] patrolled ) [11:48:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201439 MB (3% inode=25%): [11:49:51] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7294 of [[02User:Shanmugamp710]] patrolled ) [11:50:15] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User:PressRTbox10]]": Spam) [11:50:26] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User:Tinatianw10]]": Spam) [11:50:35] [[Special:Log/delete]] delete 10 * Krinkle * (deleted "[[02User:Jsoneu128210]]": Spam) [11:50:59] [[Special:Log/patrol]] patrol 10 * Krinkle * (marked revision 7280 of [[02User:IvanLanin10]] patrolled ) [11:52:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.102539/1.10, alarm hl:np_load_long=0.874023/1.55, alarm hl:mem_free=9661.000000M/500M, alarm hl:tmp_free=15084M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.102539/1.00, alarm hl:np_load_long=0.874023/1.50, alarm hl:mem_free=9661.000000M/600M, alarm hl:tmp_free=1 [11:54:23] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 68965 MB (17% inode=99%): [12:05:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9325.000000 [12:06:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:11:02] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:16:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:18:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:22:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41792 MB (10% inode=99%): [12:23:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:35:23] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 60052 MB (14% inode=99%): [12:48:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 196459 MB (3% inode=24%): [12:51:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41797 MB (10% inode=99%): [13:05:23] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58321 MB (14% inode=99%): [13:05:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10549.000000 [13:06:05] Hello all [13:06:09] Happy Pfingsten! [13:07:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:11:02] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:16:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:17:56] @replag [13:17:57] DaBPunkt: s2-user: 1d 9h 26m 3s [+0.85 s/s]; s2-user-c: 2h 58m 15s [+0.04 s/s]; s4-rr-a: 17m 9s [+0.01 s/s]; s4-user: 17m 9s [+0.01 s/s]; s5-user-c: 2h 58m 16s [+0.04 s/s]; s7-rr-a: 15s [+0.00 s/s]; s7-user: 15s [+0.00 s/s] [13:18:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:18:15] I guess the wmf-fols will rotate s4 soon [13:23:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:33:23] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1948.000000 [13:33:23] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1951.000000 [13:33:33] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1957.000000 [13:48:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 195407 MB (3% inode=24%): [13:51:38] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [13:51:38] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [13:51:38] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [13:54:33] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1959.000000 [13:54:34] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1960.000000 [13:54:34] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1960.000000 [14:01:34] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 33.000000 [14:01:34] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 33.000000 [14:01:35] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 33.000000 [14:05:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10216.000000 [14:07:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:08:27] [[Special:Log/newusers]] create 10 * Camillechidiac * (New user account) [14:10:44] 3(commented) [ACCAPP-502] tool locate article-space pages without talk pages <10https://jira.toolserver.org/browse/ACCAPP-502> (DaB.) [14:11:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:16:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:18:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:21:12] [[Special:Log/newusers]] create 10 * Mandy76 * (New user account) [14:23:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:31:46] 3(created) [MNT-1244] Switch master for s4; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1244> (DaB.) [14:42:58] @replag [14:42:59] DaBPunkt: s1-user: 48s [+0.00 s/s]; s2-user: 1d 10h 38m 5s [+0.85 s/s]; s2-user-c: 3h 18m 8s [+0.23 s/s]; s5-user-c: 3h 18m 8s [+0.23 s/s] [14:48:44] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 191253 MB (3% inode=24%): [14:49:02] DaBPunkt, there was a s4 rotation a few hours ago [14:49:16] (emergency work) [14:49:35] ah, ok you already noticed [15:00:45] 3(commented) [ACCAPP-477] Account for hosting Wikipedia-OpenStreetMap integration tools <10https://jira.toolserver.org/browse/ACCAPP-477> (DaB.) [15:05:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12807.000000 [15:06:48] Platonides: yes, the problem is that the old master is already back as slave and OUR slave uses now a slave as master… – I have to look how to fix this [15:07:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:10:13] doesn't seem too bad in this case [15:10:38] seems db51 didn't process updates the slaves missed [15:10:59] it's not the right way, but should be working until it gets fixed [15:11:13] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:15:44] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:53] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:53] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:53] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:15:53] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:53] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:16:03] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:03] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:03] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:03] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:03] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:14] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:14] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:14] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:14] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:14] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:14] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:14] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:15] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:15] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:16] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:33] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 995 MB (99% inode=99%): [15:16:33] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 995 MB (99% inode=99%): [15:16:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 74940 MB (18% inode=99%): [15:16:33] / on z-dat-s6-a is OK: DISK OK - free space: / 8281 MB (27% inode=85%): [15:16:33] Load avg. on z-dat-s6-a is OK: OK - load average: 0.25, 0.88, 1.46 [15:16:44] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 139941 MB (14% inode=99%): [15:16:44] Load avg. on z-dat-s4-a is OK: OK - load average: 0.24, 0.88, 1.46 [15:16:44] / on z-dat-s4-a is OK: DISK OK - free space: / 8281 MB (27% inode=85%): [15:16:44] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 993 MB (99% inode=99%): [15:16:44] / on z-dat-s3-a is OK: DISK OK - free space: / 8281 MB (27% inode=85%): [15:16:45] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 139941 MB (14% inode=99%): [15:16:45] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:16:46] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:16:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:17:02] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [15:17:03] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [15:17:03] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [15:17:04] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 920 MB (99% inode=99%): [15:17:04] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 87970 MB (21% inode=99%): [15:17:04] Load avg. on z-dat-s7-a is OK: OK - load average: 0.28, 0.84, 1.43 [15:17:04] / on z-dat-s7-a is OK: DISK OK - free space: / 8281 MB (27% inode=85%): [15:17:13] MySQL on z-dat-s7-a is OK: Uptime: 213707 Threads: 11 Questions: 55669531 Slow queries: 9136 Opens: 512018 Flush tables: 1 Open tables: 3614 Queries per second avg: 260.494 [15:17:13] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 148.000000 [15:17:13] MySQL slave on z-dat-s7-a is OK: Uptime: 213707 Threads: 11 Questions: 55669529 Slow queries: 9136 Opens: 512018 Flush tables: 1 Open tables: 3614 Queries per second avg: 260.494 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 225 [15:17:13] SMF on z-dat-s7-a is OK: OK - all services online [15:17:23] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:17:33] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:17:43] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:17:43] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:17:44] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:18:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:21:26] [[User:Camillechidiac]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7311&rcid=9787 * Camillechidiac * (+535) (Created page with "Camille Chidiac is a graduate of the American University of Beirut and holds a BA degree in Near East History. Chidiac is also a member of the Director’s Guild of America, He i...") [15:21:44] 3(updated) [ACCAPP-503] Running Bot on Arabic Wikipedia <10https://jira.toolserver.org/browse/ACCAPP-503> (Ciphers) [15:24:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:27:07] [[Special:Log/delete]] delete 10 * Dab * (deleted "[[02User:Camillechidiac10]]": SPAM) [15:35:44] 3(updated) [ACCAPP-514] Perform database queries across English and Telugu wiki projects for identifying quality improvement <10https://jira.toolserver.org/browse/ACCAPP-514> (Arjuna Rao Chavala) [15:37:46] 3(updated) [ACCAPP-516] account for hosting of formula drawing applets used by wikipedia-de chemistry dept. <10https://jira.toolserver.org/browse/ACCAPP-516> (Ralf Stephan) [15:42:02] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:42:03] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:43] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:03] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:03] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:03] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:23] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:43:23] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:43:33] SMTP on z-dat-s6-a is OK: SMTP OK - 0.002 sec. response time [15:43:33] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:43:54] Load avg. on z-dat-s3-a is OK: OK - load average: 0.73, 0.74, 0.97 [15:43:54] Load avg. on z-dat-s7-a is OK: OK - load average: 0.73, 0.74, 0.97 [15:43:54] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:43:54] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:43:54] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:43:54] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:45:10] DaBPunkt: why not limit page creation by new accounts? [15:45:43] 3(commented) [ACCAPP-516] account for hosting of formula drawing applets used by wikipedia-de chemistry dept. <10https://jira.toolserver.org/browse/ACCAPP-516> (Ralf Stephan) [15:46:33] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 189354 MB (3% inode=24%): [16:00:44] 3(commented) [ACCAPP-517] Account request to conduct a research on Indonesian Wikipedia quality improvement strategy <10https://jira.toolserver.org/browse/ACCAPP-517> (Ivan Lanin ) [16:05:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11912.000000 [16:07:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:11:03] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:11:23] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:17:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:18:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:24:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:30:43] 3(updated) [PATHOSCHILD-9] TemplateScript should not discard the previous this.UndoText if the text was not changed <10https://jira.toolserver.org/browse/PATHOSCHILD-9> (Helder Wiki) [16:31:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.833008/1.95, alarm hl:tmp_free=23843M/100M, alarm hl:np_load_avg=0.578125/2.0, alarm hl:mem_free=219.000000M/350M, alarm hl:available=1/0 [16:32:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:32:43] 3(updated) [PATHOSCHILD-8] Use protocol relative URLs <10https://jira.toolserver.org/browse/PATHOSCHILD-8> (Helder Wiki) [16:34:42] 3(closed) [PATHOSCHILD-8] Use protocol relative URLs <10https://jira.toolserver.org/browse/PATHOSCHILD-8> (Jesse Plamondon-Willard) [16:49:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 187161 MB (3% inode=23%): [16:49:45] 3(closed) [PATHOSCHILD-9] TemplateScript should not discard the previous this.UndoText if the text was not changed <10https://jira.toolserver.org/browse/PATHOSCHILD-9> (Jesse Plamondon-Willard) [16:52:43] 3(updated) [ACCAPP-513] Request for new account for Oxford affiliated project for analysing Wikipedia editing communities across different wiki languages <10https://jira.toolserver.org/browse/ACCAPP-513> (Ahmed Medhat) [16:56:30] hi, is there any bot operator around? if so... please PM. I've got some questions... [16:56:49] [[Special:Log/newusers]] create 10 * Jnlj * (New user account) [16:57:50] Betacommand: DaBPunkt: Yeah, could be done with rate limit maybe. Not sure if that is available in the odd version that toolserver runs (svn trunk of last year) [16:57:57] but should work [16:58:38] Krinkle: just limit it to autoconfimed accounts [17:03:27] diegogrez, what do you want? [17:03:44] Platonides: I got it [17:06:23] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9036.000000 [17:07:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:10:04] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:34] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:11:43] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [17:16:35] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:17:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:18:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:24:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:46:43] 3(commented) [OSM-17] pg_dump is too old <10https://jira.toolserver.org/browse/OSM-17> (Kolossos ) [17:49:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 186074 MB (3% inode=23%): [17:54:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44010 MB (10% inode=99%): [17:55:04] /sql on cassia is WARNING: DISK WARNING - free space: /sql 118807 MB (10% inode=99%): [17:59:04] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 77423 MB (19% inode=99%): [18:06:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9980.000000 [18:07:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:11:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:17:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:18:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:24:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:27:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.711914/1.95, alarm hl:tmp_free=23048M/100M, alarm hl:np_load_avg=0.727539/2.0, alarm hl:mem_free=274.000000M/350M, alarm hl:available=1/0 [18:31:43] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:32:42] 3(created) [PATHOSCHILD-10] Formatting of TemplateScript menus is inverted; Pathoschild's tools; Bug <10https://jira.toolserver.org/browse/PATHOSCHILD-10> (Helder Wiki) [18:41:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.589844/1.95, alarm hl:tmp_free=22976M/100M, alarm hl:np_load_avg=0.656738/2.0, alarm hl:mem_free=269.000000M/350M, alarm hl:available=1/0 [18:49:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 185518 MB (3% inode=23%): [19:06:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10062.000000 [19:07:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:11:59] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:17:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:18:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:21:18] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:22:19] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [19:24:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:41:02] [[User:Rzuwig]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7312&rcid=9790 * Rzuwig * (+32) (N) [19:42:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.703613/1.95, alarm hl:tmp_free=22568M/100M, alarm hl:np_load_avg=0.698731/2.0, alarm hl:mem_free=269.000000M/350M, alarm hl:available=1/0 [19:43:50] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:46:34] [[User:Rzuwig/common.js]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7313&rcid=9791 * Rzuwig * (+603) (N) [19:49:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 185407 MB (3% inode=23%): [19:59:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.625488/1.95, alarm hl:tmp_free=22403M/100M, alarm hl:np_load_avg=0.612793/2.0, alarm hl:mem_free=279.000000M/350M, alarm hl:available=1/0 [20:06:50] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8315.000000 [20:08:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:12:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:17:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:18:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:24:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:37:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.682617/1.95, alarm hl:tmp_free=22211M/100M, alarm hl:np_load_avg=0.724609/2.0, alarm hl:mem_free=130.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.682617/2.3, alarm hl:np_load_long=0.741211/2.5, alarm hl:cpu=59.700000/98, alarm hl:mem_free=130.000000M/200M, al [20:49:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 185194 MB (3% inode=23%): [20:54:59] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:57:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.681152/1.95, alarm hl:tmp_free=22104M/100M, alarm hl:np_load_avg=0.643555/2.0, alarm hl:mem_free=193.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.681152/2.3, alarm hl:np_load_long=0.680176/2.5, alarm hl:cpu=67.400000/98, alarm hl:mem_free=193.000000M/200M, al [21:01:00] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [21:03:15] is there a tool that can determine admin activity [21:03:25] I want to know admins that have been inactive for the past 2+ years [21:03:45] dungodung might have such a tool. [21:03:47] Or Pathoschild. [21:03:58] pathos had such a tool [21:03:58] hmm [21:04:02] stewardy probably [21:04:12] stewardry [21:04:21] and that's pathos' tool [21:04:25] yay [21:04:27] It does have an api [21:04:33] so you might be interested in that [21:04:54] It is slow on big wikis, but otherwise it works pretty well [21:05:03] it also is disabled on enwiki [21:05:14] and, I'm unsure what your definition of inactive is [21:05:23] ie, edits, admin actions (the tool offers info for both) [21:07:49] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10330.000000 [21:08:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:09:45] 3(updated) [PATHOSCHILD-10] Formatting of TemplateScript menus is inverted <10https://jira.toolserver.org/browse/PATHOSCHILD-10> (Helder Wiki) [21:12:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:13:21] http://toolserver.org/~pathoschild/stewardry/?wiki=tr.wikipedia&sysop=on&bureaucrat=on&checkuser=on&oversight=on [21:13:30] I want to check for inactive = 2 years [21:13:37] semi-active = 1 year [21:14:43] 3(commented) [PATHOSCHILD-10] Formatting of TemplateScript menus is inverted <10https://jira.toolserver.org/browse/PATHOSCHILD-10> (Helder Wiki) [21:17:49] / on willow is WARNING: DISK WARNING - free space: / 21994 MB (20% inode=99%): [21:17:49] /tmp on willow is WARNING: DISK WARNING - free space: / 21994 MB (20% inode=99%): [21:18:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:19:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:20:49] / on willow is OK: DISK OK - free space: / 31093 MB (29% inode=99%): [21:20:50] /tmp on willow is OK: DISK OK - free space: / 31093 MB (29% inode=99%): [21:22:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.682129/1.95, alarm hl:tmp_free=31023M/100M, alarm hl:np_load_avg=0.723144/2.0, alarm hl:mem_free=240.000000M/350M, alarm hl:available=1/0 [21:24:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:29:00] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [21:32:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.739746/1.95, alarm hl:tmp_free=40102M/100M, alarm hl:np_load_avg=0.746582/2.0, alarm hl:mem_free=332.000000M/350M, alarm hl:available=1/0 [21:49:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 185061 MB (3% inode=23%): [21:58:09] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.655273/1.95, alarm hl:tmp_free=39960M/100M, alarm hl:np_load_avg=0.643555/2.0, alarm hl:mem_free=237.000000M/350M, alarm hl:available=1/0 [22:01:19] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:07:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10223.000000 [22:08:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.817871/1.95, alarm hl:tmp_free=39894M/100M, alarm hl:np_load_avg=0.850586/2.0, alarm hl:mem_free=129.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.817871/2.3, alarm hl:np_load_long=0.779785/2.5, alarm hl:cpu=71.400000/98, alarm hl:mem_free=129.000000M/200M, al [22:08:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:12:29] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:18:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:19:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:22:36] [[Special:Log/newusers]] create 10 * Kirento114 * (New user account) [22:25:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:27:23] nacht ts [22:30:09] [[Special:Log/newusers]] create 10 * Ellajqp8713 * (New user account) [22:32:20] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:35:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.697754/1.95, alarm hl:tmp_free=39727M/100M, alarm hl:np_load_avg=0.733398/2.0, alarm hl:mem_free=136.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.697754/2.3, alarm hl:np_load_long=0.743652/2.5, alarm hl:cpu=57.800000/98, alarm hl:mem_free=136.000000M/200M, al [22:47:42] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:50:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 184974 MB (3% inode=23%): [23:04:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.865234/1.95, alarm hl:tmp_free=39546M/100M, alarm hl:np_load_avg=0.852539/2.0, alarm hl:mem_free=155.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.865234/2.3, alarm hl:np_load_long=0.743164/2.5, alarm hl:cpu=56.500000/98, alarm hl:mem_free=155.000000M/200M, al [23:08:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7691.000000 [23:09:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:10:00] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [23:12:50] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:18:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:19:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:25:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:50:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 184603 MB (3% inode=23%): [23:56:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.