[00:13:49] <tsnag>	 SMF on wolfsbane is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[00:27:38] <tsnag>	 Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat  
[00:27:48] <tsnag>	 Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat  
[00:27:48] <tsnag>	 Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat  
[00:28:18] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[00:29:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[00:29:48] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[00:29:49] <tsnag>	 SMF on ortelius is CRITICAL: ERROR - maintenance:  svc:/application/sge/execd:toolserver  
[00:29:49] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[00:30:49] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:30:49] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[00:30:49] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[00:31:38] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[00:32:08] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559828 MB (10% inode=49%):  
[00:38:38] <tsnag>	 Sun Grid Engine execd on ortelius is CRITICAL: short-sol@ortelius in unknown state: medium-sol@ortelius in unknown state  
[00:38:59] <tsnag>	 Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in unknown state: medium-sol@wolfsbane in unknown state  
[00:38:59] <tsnag>	 Sun Grid Engine execd on willow is CRITICAL: medium-sol@willow in unknown state: longrun-sol@willow in unknown state  
[00:45:59] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[00:46:39] <tsnag>	 Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK  
[00:46:59] <tsnag>	 Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane disabled: medium-sol@wolfsbane disabled  
[00:47:00] <tsnag>	 SMF on ortelius is OK: OK - all services online  
[00:47:00] <tsnag>	 SMF on wolfsbane is OK: OK - all services online  
[01:16:02] <reba>	 DaB. * Re: [Toolserver-announce] [Toolserver-l] Maintenance at Wednesday
[01:28:18] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[01:30:59] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[01:30:59] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:31:00] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[01:31:38] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[01:32:09] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 550135 MB (10% inode=48%):  
[02:08:23] <DaBPunkt>	 nacht ts
[02:20:59] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.146973/1.8, alarm hl:np_load_avg=1.029297/2.3, alarm hl:mem_free=289.000000M/300M, alarm hl:available=1/0  
[02:22:59] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[02:28:30] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[02:28:59] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.967285/1.8, alarm hl:np_load_avg=0.937500/2.3, alarm hl:mem_free=288.000000M/300M, alarm hl:available=1/0  
[02:30:59] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[02:30:59] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:30:59] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[02:31:38] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[02:32:18] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547535 MB (10% inode=48%):  
[02:34:56] <msgbot>	 3(created) [ACCAPP-476] To run a Bot for scheduled work and routine event driven tasks on Marathi Wiipedia (mr.wiki).; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-476>  (Rahuldeshmukh101)
[03:29:30] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[03:30:59] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[03:30:59] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:30:59] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[03:31:48] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[03:33:18] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547592 MB (10% inode=48%):  
[04:23:09] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.907227/1.8, alarm hl:np_load_avg=1.004883/2.3, alarm hl:mem_free=280.000000M/300M, alarm hl:available=1/0  
[04:24:09] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[04:29:48] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[04:31:09] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:31:09] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[04:31:09] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[04:32:09] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[04:33:39] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547432 MB (10% inode=48%):  
[04:38:09] <tsnag>	 Load avg. on cassia is WARNING: WARNING - load average: 17.14, 12.51, 9.11  
[04:53:08] <tsnag>	 Load avg. on cassia is OK: OK - load average: 9.30, 14.28, 13.99  
[05:22:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.985840/1.8, alarm hl:np_load_avg=1.071289/2.3, alarm hl:mem_free=283.000000M/300M, alarm hl:available=1/0  
[05:26:19] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[05:27:48] <tsnag>	 SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[05:28:40] <tsnag>	 SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[05:29:48] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[05:31:18] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[05:31:18] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:31:18] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[05:33:07] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[05:33:39] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547387 MB (10% inode=48%):  
[06:12:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.763672/1.8, alarm hl:np_load_avg=1.634766/2.3, alarm hl:mem_free=299.000000M/300M, alarm hl:available=1/0  
[06:13:19] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[06:30:09] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[06:31:19] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:31:19] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[06:31:20] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[06:33:08] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[06:33:19] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.856445/1.8, alarm hl:np_load_avg=1.739746/2.3, alarm hl:mem_free=766.000000M/300M, alarm hl:available=1/0  
[06:33:39] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547333 MB (10% inode=48%):  
[06:51:28] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 19.20, 16.35, 13.69  
[06:52:27] <tsnag>	 Load avg. on willow is OK: OK - load average: 11.76, 14.71, 13.27  
[06:56:28] <tsnag>	 Load avg. on willow is WARNING: WARNING - load average: 20.64, 17.07, 14.45  
[07:30:20] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[07:31:19] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[07:31:19] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:31:19] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[07:33:09] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[07:33:49] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547232 MB (10% inode=48%):  
[08:05:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.108399/1.8, alarm hl:np_load_avg=1.330078/2.3, alarm hl:mem_free=279.000000M/300M, alarm hl:available=1/0  
[08:08:34] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[08:30:42] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[08:31:33] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:31:33] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[08:31:34] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[08:32:36] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.945312/1.8, alarm hl:np_load_avg=1.562012/2.3, alarm hl:mem_free=392.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.945312/1.9, alarm hl:np_load_long=1.389160/2.25, alarm hl:mem_free=392.000000M/200M, alarm hl:available=1/0  
[08:33:33] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[08:33:33] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[08:34:43] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 547135 MB (10% inode=48%):  
[08:49:33] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.102539/1.8, alarm hl:np_load_avg=1.186524/2.3, alarm hl:mem_free=296.000000M/300M, alarm hl:available=1/0  
[09:31:41] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[09:32:33] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[09:32:33] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:32:33] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[09:33:33] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[09:34:53] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546959 MB (10% inode=48%):  
[09:41:58] <msgbot>	 3(created) [ACCAPP-477] Account for hosting Wikipedia-OpenStreetMap integration tools; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-477> 
[10:21:53] <msgbot>	 3(created) [TS-1329] Page not listing unassessed articles; Toolserver: Webserver: General/Unknown; Bug <10https://jira.toolserver.org/browse/TS-1329>  (Shagnik Saraswatui)
[10:31:42] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[10:32:33] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[10:32:33] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:32:33] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[10:34:33] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[10:35:32] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546819 MB (10% inode=48%):  
[10:46:53] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.5% (271008 kB) free!  
[10:47:53] <tsnag>	 Free Memory on damiana is OK: OK - 7.5% (313220 kB) free.  
[11:01:12] <tsnag>	 RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:11:02] <tsnag>	 RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[11:18:45] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41427 MB (10% inode=99%):  
[11:31:54] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[11:32:44] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[11:32:45] <tsnag>	 /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24359 MB (5% inode=99%):  
[11:32:45] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:32:45] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[11:34:46] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[11:35:53] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546706 MB (10% inode=48%):  
[11:36:12] <tsnag>	 RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:45:06] <DaBPunkt>	 Hello all
[11:45:52] <msgbot>	 3(commented) [MNT-1222] maintenance Wed 14. March <10https://jira.toolserver.org/browse/MNT-1222>  (DaB.)
[11:46:50] <Josh_Parris>	 Howdy Dab
[11:50:54] <msgbot>	 3(resolved) [MNT-1222] maintenance Wed 14. March <10https://jira.toolserver.org/browse/MNT-1222>  (DaB.)
[11:52:53] <msgbot>	 3(commented) [TS-1326] Database corruption in enwiki_p on thyme <10https://jira.toolserver.org/browse/TS-1326>  (DaB.)
[11:56:44] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 65548 MB (16% inode=99%):  
[12:32:44] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[12:32:52] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[12:33:44] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:33:44] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[12:35:44] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[12:36:03] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546608 MB (10% inode=48%):  
[12:41:44] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43292 MB (10% inode=99%):  
[13:06:03] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.7% (280348 kB) free!  
[13:21:12] <tsnag>	 Free Memory on damiana is OK: OK - 7.1% (298428 kB) free.  
[13:23:44] <tsnag>	 /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24461 MB (5% inode=99%):  
[13:24:12] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 6.9% (290504 kB) free!  
[13:30:03] <tsnag>	 Free Memory on damiana is OK: OK - 7.1% (295900 kB) free.  
[13:32:44] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[13:33:12] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[13:33:53] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:33:54] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[13:35:44] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[13:36:03] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546490 MB (10% inode=48%):  
[13:50:03] <tsnag>	 Free Memory on damiana is WARNING: WARNING - 7.0% (291120 kB) free!  
[13:53:13] <tsnag>	 Free Memory on damiana is OK: OK - 7.4% (308952 kB) free.  
[14:21:44] <tsnag>	 /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42504 MB (10% inode=99%):  
[14:32:52] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[14:33:22] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[14:33:52] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:34:53] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[14:35:44] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[14:36:13] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546365 MB (10% inode=48%):  
[14:42:43] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.747559/1.8, alarm hl:np_load_avg=0.738281/2.3, alarm hl:mem_free=294.000000M/300M, alarm hl:available=1/0  
[14:49:44] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[14:58:44] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.817383/1.8, alarm hl:np_load_avg=0.827148/2.3, alarm hl:mem_free=281.000000M/300M, alarm hl:available=1/0  
[15:12:53] <tsnag>	 /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24373 MB (5% inode=99%):  
[15:14:00] <msgbot>	 3(moved) [TS-1330] Account re-enablling of account "josh" <10https://jira.toolserver.org/browse/TS-1330>  (DaB.)
[15:16:54] <msgbot>	 3(resolved) [TS-1330] Account re-enablling of account "josh" <10https://jira.toolserver.org/browse/TS-1330>  (DaB.)
[15:32:53] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[15:33:22] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[15:33:53] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:34:53] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[15:35:53] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[15:36:12] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546249 MB (10% inode=48%):  
[15:39:31] <tsnag>	 fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds  
[15:41:53] <msgbot>	 3(commented) [ACCAPP-449] Account Approval for the toolserver withing the RENDER project <10https://jira.toolserver.org/browse/ACCAPP-449>  (Gerrit Holz)
[15:46:54] <msgbot>	 3(commented) [ACCAPP-449] Account Approval for the toolserver withing the RENDER project <10https://jira.toolserver.org/browse/ACCAPP-449>  (DaB.)
[15:47:21] <tsnag>	 SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:47:22] <tsnag>	 SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:47:23] <tsnag>	 SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:47:23] <tsnag>	 SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:47:23] <tsnag>	 SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[15:47:42] <tsnag>	 RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:48:13] <tsnag>	 SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[15:48:14] <tsnag>	 SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[15:48:14] <tsnag>	 SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[15:48:22] <tsnag>	 SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[15:48:22] <tsnag>	 SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[15:48:32] <tsnag>	 RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[15:52:22] <tsnag>	 fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 13.716 second response time  
[15:58:52] <CIA-91>	 toolserver: 03dab * r1141 10/trunk/TSStatus/WEB-INF/src/org/toolserver/tsstatus/TSStatusServlet.java: 
[15:58:52] <CIA-91>	 toolserver: -Improved xml-cache-handling
[15:58:52] <CIA-91>	 toolserver: -Completed slave-mode.
[16:04:43] <tsnag>	 fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds  
[16:06:52] <chicocvenancio>	 How long is the usual wait time for new account requests?
[16:15:31] <tsnag>	 fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 14.128 second response time  
[16:17:43] <DaBPunkt>	 chicocvenancio: few days until several weeks (depend on how busy am I)
[16:17:53] <shantanoo>	 hi DaBPunkt 
[16:17:59] <DaBPunkt>	 but if it is urgend, you can ping me rom time to time
[16:18:55] <shantanoo>	 if i want to help in running the queries requested from bugzilla, is there any pre-req for doing that?
[16:19:50] <DaBPunkt>	 shantanoo: yes, we have no bugzilla ;-).
[16:20:47] <shantanoo>	 DaBPunkt: oh. then i suppose i misinterpreted some email communication :)
[16:21:04] <DaBPunkt>	 shantanoo: but no, there is no pre-req other than you have to know mysql
[16:22:05] <shantanoo>	 one of the reason i am interested in it is because i can learn little more about getting statistics from the requests :)
[16:22:36] <shantanoo>	 and while i learn, it can be helpful for others...
[16:22:51] <shantanoo>	 is it ok do so?
[16:23:07] <DaBPunkt>	 shantanoo: sure
[16:23:24] <DaBPunkt>	 as long as you don't kill our database--servers ;)
[16:23:24] <shantanoo>	 DaBPunkt: so where do i start?
[16:23:40] <shantanoo>	 should i be subscribed to some mailing list?
[16:23:42] <DaBPunkt>	 do you have an ts-account already?
[16:23:54] <shantanoo>	 yes.
[16:24:01] <shantanoo>	 i do have ts account.
[16:24:34] <DaBPunkt>	 shantanoo: https://wiki.toolserver.org/view/DBQ
[16:25:46] * shantanoo  checking link

[16:25:53] <tsnag>	 /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 135912 MB (33% inode=99%):  
[16:32:54] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[16:33:22] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[16:33:54] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:35:02] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[16:35:54] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[16:36:13] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 546090 MB (10% inode=48%):  
[16:44:53] <tsnag>	 Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.662109/1.8, alarm hl:np_load_avg=0.749512/2.3, alarm hl:mem_free=206.000000M/300M, alarm hl:available=1/0  
[16:46:53] <tsnag>	 Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK  
[16:57:46] <CIA-91>	 toolserver: 03dab * r1142 10/trunk/TSStatus/WEB-INF/src/org/toolserver/tsstatus/ShowPlain.java: -Display Slave-Data here too.
[17:08:54] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1922.000000  
[17:20:53] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1922.000000  
[17:31:36] <addshore>	 anyone here amazing at using the toolserver scheduling system? :)
[17:33:22] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[17:33:54] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[17:33:54] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:35:03] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[17:36:13] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545924 MB (10% inode=48%):  
[17:36:53] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[17:44:54] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000  
[17:51:54] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3639.000000  
[18:33:32] <tsnag>	 FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk  
[18:34:02] <tsnag>	 SMF on willow is CRITICAL: ERROR - maintenance:  svc:/network/puppetmasterd:default  
[18:34:03] <tsnag>	 SMF on turnera is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:36:03] <tsnag>	 RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[18:36:23] <tsnag>	 /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 545781 MB (10% inode=48%):  
[18:37:02] <tsnag>	 SMF on damiana is CRITICAL: ERROR -  offline:  svc:/system/cluster/scsymon-srv:default  
[18:44:53] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6755.000000  
[18:51:52] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7010.000000  
[19:00:07] <Alchimista>	 error: user "alchimista" not found in password database oO
[19:00:26] <Alchimista>	 i'm getting this error ^ when using qstat
[19:10:53] <msgbot>	 3(assigned) [ACCAPP-477] Account for hosting Wikipedia-OpenStreetMap integration tools <10https://jira.toolserver.org/browse/ACCAPP-477>  (Kolossos )
[19:40:52] <CIA-91>	 toolserver: 03dab * r1143 10/trunk/TSStatus/WEB-INF/ (4 files in 2 dirs): 
[19:40:52] <CIA-91>	 toolserver: -Add API-interface.
[19:40:52] <CIA-91>	 toolserver: -Clean-UP.
[19:46:10] <DaBPunkt>	 I will update status.toolserver.org now. Should be bakc in 1 minute
[19:47:51] <DaBPunkt>	 ok, done
[19:51:23] <Nemo_bis>	 DaBPunkt, can SGE be used?
[19:51:30] <DaBPunkt>	 yes
[19:51:43] <Nemo_bis>	 and my pocess won't be killed? :)
[19:51:48] <Nemo_bis>	 *processes
[19:52:00] <DaBPunkt>	 pardon?
[19:52:08] <Nemo_bis>	 no new updates?
[19:52:18] <Nemo_bis>	 just to avoid wasting CPU time
[19:53:09] <DaBPunkt>	 not until next maintenance-window
[19:54:56] <Nemo_bis>	 oki
[19:55:04] <Nemo_bis>	 I guess I have some days then
[20:06:02] <reba>	 DaB. * Re: [Toolserver-l] Improvment of status.toolserver.org
[20:08:18] <Daniel_krank>	 DaBPunkt: hi!
[20:08:36] <DaBPunkt>	 Daniel_krank: Ab ins Bett!
[20:08:50] <DaBPunkt>	 Wer krank ist, hat vor dem Computer nix verloren ;)
[20:08:51] <Daniel_krank>	 hehe
[20:08:58] <Daniel_krank>	 ich kann ich mit comupter ins bett :)
[20:09:08] <DaBPunkt>	 ach, du auch..
[20:09:13] <Daniel_krank>	 :P
[20:09:27] <DaBPunkt>	 was fehlt dir?
[20:09:38] <Daniel_krank>	 erkältung. watte im kopf.
[20:09:42] <Daniel_krank>	 geht aber schon wieder besser
[20:09:50] <Daniel_krank>	 bin die woche krankgeschrieben
[20:09:51] <Daniel_krank>	 naja
[20:09:53] <Daniel_krank>	 was'n mit puppet los? meine inbox läuft voll...
[20:10:35] <DaBPunkt>	 Daniel_krank: "ifconfig: status: SIOCGLIFFLAGS: e1000g0:8: no such interface" meinst du?
[20:10:40] <Daniel_krank>	 ja
[20:10:44] <Daniel_krank>	 ein paar hundert mails...
[20:10:48] <DaBPunkt>	 das ist schon immer so
[20:10:58] <Daniel_krank>	 aber ich bekomme nicht schon immer ständig mails darüber
[20:11:25] <DaBPunkt>	 Daniel_krank: ich habe vor kurzem was am mail-server korrigiert, vielleciht komemn die emails deswegen jetzt durch
[20:11:28] <Daniel_krank>	 bekommst du die nicht?
[20:11:38] <Daniel_krank>	 die gehen an root@toolserver.org
[20:11:43] <DaBPunkt>	 ich hab dafür einen Ordner…
[20:11:53] <Daniel_krank>	 hrmpf
[20:12:01] <Daniel_krank>	 das sind dann aber doch ein paar zu viele
[20:12:19] <Daniel_krank>	 kann man das filtern?
[20:12:23] <Daniel_krank>	  das nervt schon ganz schön
[20:12:32] <Daniel_krank>	 ok, ich kann die auch automatisch löschen
[20:12:46] <Daniel_krank>	 aber diese art von müll führt dazu, dass man "echte" fehler nicht bemerkt
[20:12:50] <DaBPunkt>	 Daniel_krank: das übersteigt zur Zeit noch meine puppet-fähigkeiten
[20:13:09] <Daniel_krank>	 im cron einfach ein grep -v dahinter :)
[20:13:26] <DaBPunkt>	 wäre 'ne idee
[20:13:48] <Daniel_krank>	 oder andersrum greppen... der header kommt ja immer
[20:14:00] <Daniel_krank>	 auch schön: /opt/local/bin/status.sh[3]: scstat:  not found
[20:14:15] <Daniel_krank>	 "reboot_when_memory_low"  <--- wtf?
[20:17:12] <DaBPunkt>	 Daniel_krank: eine Schutzschaltung. Sie rebootet eine HA-node bevor ihr Speicher zu stark zur Neige gehen
[20:22:00] <Duesentrieb>	 DaBPunkt: aber anscheinend tun sie genau das nicht
[20:22:39] <DaBPunkt>	 Duesentrieb: doch, tut sie schon. Die kurzen Schluckaufs sind die reboots
[20:23:27] <Duesentrieb>	 und die meldung ist nur 'ne warnung? und der reboot geht trotzdem durch
[20:23:32] <Duesentrieb>	 wieso passiert das mehrmals täglich?
[20:24:15] <Duesentrieb>	 vielleicht sollte ich root@toolserver.org gleich nach /dev/null schicken :(
[20:24:41] <DaBPunkt>	 http://munin.toolserver.org/Miscellaneous/damiana/uptime.html von wegen mehrmals täglich…
[20:27:49] <Duesentrieb>	 DaBPunkt: ok, dann hab ich das mit den Schluckaufs falsch verstanden. 
[20:27:58] <Duesentrieb>	 hm... ich bekomme alle 5 minuten 'ne mail von dem script
[20:28:01] <Duesentrieb>	 das nervt schon ganz schön
[20:28:10] <DaBPunkt>	 ich schaue gerade wo da das problem ist
[20:28:58] <DaBPunkt>	 sollte gefixt sein
[20:52:55] <msgbot>	 3(created) [MNT-1223] Remove old kill-queries-cron from db-hosts; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1223>  (DaB.)
[20:56:52] <msgbot>	 3(resolved) [MNT-1223] Remove old kill-queries-cron from db-hosts <10https://jira.toolserver.org/browse/MNT-1223>  (DaB.)
[21:02:09] <Duesentrieb>	 DaBPunkt: danke, jetzt ist es nur noch puppet :)
[21:02:38] <DaBPunkt>	 Duesentrieb: das sollte sich auch bald erledigen.
[21:02:50] <Duesentrieb>	 yay :)
[22:24:55] <DaBPunkt>	 there is a problem with the sql-toolserver-mysql-instance. I look for it
[22:55:44] <Danny_B|webchat>	 DaBPunkt:  NFS server ha-nfs.esi not responding still trying
[22:55:55] <DaBPunkt>	 yes, I know. Wait a moment
[22:57:24] <Danny_B|webchat>	 :-(((
[22:57:27] <Krinkle>	 willow isn't responding either, bots on it timed out
[22:57:36] <Danny_B|webchat>	 i hope it was not interrupted
[22:58:13] <Danny_B|webchat>	 nightshade away, willow away...
[23:00:44] <Seahorse>	 willow is in a deep state of depression because its friend nightshade is dead. Now there is no place to run anything.
[23:01:32] <Seahorse>	 Kind of annoying the way willow is pretty unreliable lately
[23:01:48] <Krinkle>	 it's back up already
[23:02:20] <Seahorse>	 It was still an inconvenience since everyone has to restart their stuff..
[23:03:20] <Danny_B|webchat>	 back up, but not my irssi in screen  :-/
[23:03:33] <Danny_B|webchat>	 and idk how to get it running
[23:04:24] <Seahorse>	 "irssi"
[23:05:06] <Seahorse>	 Danny_B|webchat, they allow us to run irssi for casual purposes? hmm..
[23:10:54] <jem->	 Problems again...
[23:11:21] <DaBPunkt>	 should be fixed now
[23:16:49] <jem->	 Yes, it's Ok again :)
[23:20:11] <DaBPunkt>	 nfs can be away for a short time again now
[23:20:50] <Danny_B|webchat>	 yes it is
[23:21:36] <jem->	 And back again
[23:21:50] <DaBPunkt>	 and back
[23:22:23] <DaBPunkt>	 at least I found out why nfs was not HA
[23:22:34] <DaBPunkt>	 but still have problems with sql-toolserver
[23:22:55] <Danny_B|webchat>	 and again away
[23:23:08] <DaBPunkt>	 not here
[23:23:18] <DaBPunkt>	 which server?
[23:24:51] <jem->	 willow seems ok for now