[00:05:16] nacht ts [00:27:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:31:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:33:05] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:33:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49223 MB (5% inode=99%): [01:28:06] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:31:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:33:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49046 MB (5% inode=99%): [01:33:16] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:40:56] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.534180/1.75, alarm hl:np_load_avg=0.468750/2.00, alarm hl:mem_free=273.000000M/300M [01:41:56] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:47:46] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:16] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:28:06] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:31:57] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:33:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48906 MB (5% inode=99%): [02:33:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:41:06] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:06] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:06] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:56] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:41:56] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:41:57] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:28:06] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:32:06] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:33:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:34:17] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48760 MB (4% inode=99%): [03:36:23] 3(commented) [DBQ-175] checking cellistbot's it.wiki edits <10https://jira.toolserver.org/browse/DBQ-175> (madman) [03:36:24] 3(assigned) [DBQ-175] checking cellistbot's it.wiki edits <10https://jira.toolserver.org/browse/DBQ-175> (madman) [03:40:06] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.490723/1.75, alarm hl:np_load_avg=0.492676/2.00, alarm hl:mem_free=238.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.490723/1.50, alarm hl:np_load_long=0.498047/1.75, alarm hl:mem_free=238.000000M/250M [03:43:06] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:13:06] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.692871/1.75, alarm hl:np_load_avg=0.619141/2.00, alarm hl:mem_free=215.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.692871/1.50, alarm hl:np_load_long=0.534668/1.75, alarm hl:mem_free=215.000000M/250M [04:14:06] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:27:14] [[Special:Log/newusers]] create 10 * Moneymarketrates3 * (New user account) [04:28:06] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:33:06] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:33:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:34:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48632 MB (4% inode=99%): [05:28:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:33:06] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:33:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:34:27] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48499 MB (4% inode=99%): [05:37:45] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:38:16] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:16] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:16] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:16] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:36] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:38:56] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [05:38:56] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [05:39:06] MySQL on z-dat-s3-a is OK: Uptime: 4287570 Threads: 15 Questions: 4786186143 Slow queries: 359162 Opens: 61796726 Flush tables: 2 Open tables: 16384 Queries per second avg: 1116.293 [05:39:06] MySQL slave on z-dat-s3-a is OK: Uptime: 4287570 Threads: 14 Questions: 4786186146 Slow queries: 359162 Opens: 61796726 Flush tables: 2 Open tables: 16384 Queries per second avg: 1116.293 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75 [05:39:06] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:39:06] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:39:06] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:39:06] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:25:39] [[Special:Log/newusers]] create 10 * Arlenemacias * (New user account) [06:28:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:33:06] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:34:26] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:35:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49335 MB (5% inode=99%): [07:29:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:33:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:34:38] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:35:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49228 MB (5% inode=99%): [08:13:17] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 26275 MB (2% inode=99%): [08:29:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:33:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:35:27] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49104 MB (5% inode=99%): [08:35:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:49:05] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.499512/1.75, alarm hl:np_load_avg=0.532715/2.00, alarm hl:mem_free=242.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.499512/1.50, alarm hl:np_load_long=0.509277/1.75, alarm hl:mem_free=242.000000M/250M [08:56:06] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:29:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:33:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:35:27] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48984 MB (5% inode=99%): [09:35:46] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:13:07] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:13:36] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:30:26] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:34:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:35:22] 3(commented) [DRTRIGON-112] subster_irc bot forgets wiki login when accessing other mediawiki project <10https://jira.toolserver.org/browse/DRTRIGON-112> (drtrigon) [10:36:27] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48847 MB (5% inode=99%): [10:36:46] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:55:27] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:27] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:36] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:36] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:18] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:56:18] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:56:26] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [10:56:26] SMTP on hyacinth is OK: SMTP OK - 0.003 sec. response time [11:12:28] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.381348/1.75, alarm hl:np_load_avg=0.363281/2.00, alarm hl:mem_free=235.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.381348/1.50, alarm hl:np_load_long=0.335938/1.75, alarm hl:mem_free=235.000000M/250M [11:15:27] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [11:30:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:34:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:36:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48712 MB (4% inode=99%): [11:36:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:30:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:34:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:36:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48568 MB (4% inode=99%): [12:36:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:47:38] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.897461/1.00, alarm hl:np_load_long=0.809570/1.50, alarm hl:mem_free=23001.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.897461/1.10, alarm hl:np_load_long=0.809570/1.75, alarm hl:mem_free=23001.000000M/300M [12:48:37] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [13:02:46] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [13:30:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:32:07] [[User:Sbharti]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6588&rcid=8674 * Sbharti * (+57) (Created page with "Shaurabh Bharti. pls contact at sbharti-at-gmail-dot-com") [13:33:03] [[User:Sbharti]] ! 10https://wiki.toolserver.org/w/index.php?diff=6589&oldid=6588&rcid=8675 * Sbharti * (+72) () [13:34:47] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:34:54] @replag [13:34:55] DaB|Busy: s2-pri: 10s [-0.00 s/s]; s3-rr: 29s [-0.00 s/s]; s3-user: 29s [-0.00 s/s] [13:36:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48412 MB (4% inode=99%): [13:36:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:02:48] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [14:30:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:34:48] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:36:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48254 MB (4% inode=99%): [14:37:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:03:09] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [15:12:56] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.345703/1.75, alarm hl:np_load_avg=0.400879/2.00, alarm hl:mem_free=241.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.345703/1.50, alarm hl:np_load_long=0.362793/1.75, alarm hl:mem_free=241.000000M/250M [15:13:17] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:37] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:37] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:47] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:56] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:57] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:13:57] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:17] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:17] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3471 MB (99% inode=99%): [15:14:27] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:14:27] NTP on hyacinth is OK: NTP OK: Offset -0.001229 secs [15:14:48] SMTP on hyacinth is OK: SMTP OK - 0.161 sec. response time [15:14:48] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:15:07] SMTP on z-dat-s4-a is OK: SMTP OK - 0.160 sec. response time [15:15:08] SMTP on z-dat-s6-a is OK: SMTP OK - 0.066 sec. response time [15:21:55] * hexmode is having trouble with mysql [15:22:36] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:48] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:23:27] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:30:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:31:58] hrm... looks like I can no longer connect to sql-s3-rr [15:34:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:35:37] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:57] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:58] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:58] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:48] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:36:48] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:36:48] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:36:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49021 MB (5% inode=99%): [15:37:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:04:08] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [16:12:47] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:12:58] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:59] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:59] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:28] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:38] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:38] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:47] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [16:13:48] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [16:13:48] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:48] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:48] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:13:57] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 3541 MB (99% inode=99%): [16:13:57] MySQL on z-dat-s6-a is OK: Uptime: 5592588 Threads: 6 Questions: 1087831944 Slow queries: 376314 Opens: 11109929 Flush tables: 2 Open tables: 2869 Queries per second avg: 194.513 [16:13:57] MySQL slave on z-dat-s6-a is OK: Uptime: 5592588 Threads: 5 Questions: 1087831945 Slow queries: 376314 Opens: 11109929 Flush tables: 2 Open tables: 2869 Queries per second avg: 194.513 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 217 [16:14:07] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3521 MB (99% inode=99%): [16:14:17] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3492 MB (99% inode=99%): [16:14:17] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 123124 MB (30% inode=99%): [16:14:17] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3487 MB (99% inode=99%): [16:14:26] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:14:26] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:14:47] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:14:47] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:14:47] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:30:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:34:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:36:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48847 MB (5% inode=99%): [16:37:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:04:19] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [17:30:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:35:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:36:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48669 MB (4% inode=99%): [17:37:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:04:20] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [18:13:08] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.486328/1.75, alarm hl:np_load_avg=0.451172/2.00, alarm hl:mem_free=299.000000M/300M [18:14:07] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [18:18:30] [[GeoHack]] ! 10https://wiki.toolserver.org/w/index.php?diff=6590&oldid=5798&rcid=8676 * 80.239.243.195 * (-130) (/* language */ ) [18:30:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:36:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48490 MB (4% inode=99%): [18:36:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:37:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:04:19] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [19:22:58] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=3.467773/1.00, alarm hl:np_load_long=0.949219/1.50, alarm hl:mem_free=22666.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=3.467773/1.10, alarm hl:np_load_long=0.949219/1.75, alarm hl:mem_free=22666.000000M/300M [19:25:58] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [19:31:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:37:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48312 MB (4% inode=99%): [19:37:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:37:28] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:45:08] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.628906/1.00, alarm hl:np_load_long=1.315430/1.50, alarm hl:mem_free=21742.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.628906/1.10, alarm hl:np_load_long=1.315430/1.75, alarm hl:mem_free=21742.000000M/300M [20:04:28] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [20:13:47] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 27087 MB (2% inode=99%): [20:31:17] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:37:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:37:28] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:37:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48135 MB (4% inode=99%): [21:04:28] Sun Grid Engine execd on nightshade is CRITICAL: all.q@nightshade in error state: QERROR as result of job 1517896s failure [21:09:30] cleared this error [21:09:37] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [21:31:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:37:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:37:38] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:37:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48896 MB (5% inode=99%): [21:47:24] 3(commented) [TS-1273] Add JIRA category for unblock project <10https://jira.toolserver.org/browse/TS-1273> (DaB.) [21:57:20] 3(commented) [TS-1273] Add JIRA category for unblock project <10https://jira.toolserver.org/browse/TS-1273> (Brett Reynolds) [22:31:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:37:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:37:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:38:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48724 MB (4% inode=99%): [23:07:28] nacht ts [23:28:19] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:29:08] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:31:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:38:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48570 MB (4% inode=99%): [23:38:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:38:39] SSH on nightshade.mgmt is CRITICAL: Server answer: