[00:02:44] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.142090/2.00, alarm hl:mem_free=158.000000M/250M [00:03:14] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.657226/1.50, alarm hl:mem_free=1096.000000M/100M [00:05:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:07:31] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [00:07:33] zzz [00:12:51] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.078125/2.00, alarm hl:mem_free=144.000000M/250M [00:21:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:21:51] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 255112 MB (4% inode=31%): [00:22:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68284 MB (7% inode=99%): [00:22:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:22:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:23:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:07:16] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.176758/2.00, alarm hl:mem_free=231.000000M/250M [01:07:58] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:11:29] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.197754/2.00, alarm hl:mem_free=136.000000M/250M [01:12:29] Load avg. on nightshade is WARNING: WARNING - load average: 15.11, 13.03, 11.02 [01:12:39] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.641601/1.50, alarm hl:mem_free=958.000000M/100M [01:13:30] Load avg. on nightshade is OK: OK - load average: 12.64, 12.68, 11.02 [01:17:40] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [01:22:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 257811 MB (4% inode=31%): [01:22:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68155 MB (6% inode=99%): [01:22:49] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.628418/1.50, alarm hl:mem_free=981.000000M/100M [01:22:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:22:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:22:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:38:21] 3(created) [ACCAPP-426] TjBot's request on ToolServer for Interwiki from id.wikipedia; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-426> (Moel) [01:45:29] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.975098/2.00, alarm hl:mem_free=151.000000M/250M [01:47:30] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:52:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:02:49] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.500000/1.50, alarm hl:mem_free=1306.000000M/100M [02:03:39] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [02:12:51] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.528320/1.50, alarm hl:mem_free=259.000000M/100M [02:13:50] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:16:50] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.566895/1.50, alarm hl:mem_free=371.000000M/100M [02:22:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 257148 MB (4% inode=31%): [02:23:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68032 MB (6% inode=99%): [02:23:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:23:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:23:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:29:50] Load avg. on nightshade is WARNING: WARNING - load average: 15.64, 13.19, 11.42 [02:31:00] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.828613/1.50, alarm hl:mem_free=1088.000000M/100M [02:32:51] Load avg. on nightshade is OK: OK - load average: 11.07, 13.84, 12.13 [02:35:09] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [02:48:09] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.625000/1.50, alarm hl:mem_free=1057.000000M/100M [02:52:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:06:50] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.040039/2.00, alarm hl:mem_free=171.000000M/250M [03:12:50] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:22:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 257038 MB (4% inode=31%): [03:23:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:23:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:23:10] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:23:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67881 MB (6% inode=99%): [03:24:09] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.543457/1.50, alarm hl:mem_free=1043.000000M/100M [03:29:50] Load avg. on nightshade is WARNING: WARNING - load average: 14.70, 15.37, 12.90 [03:32:50] Load avg. on nightshade is OK: OK - load average: 12.04, 14.62, 13.13 [03:37:10] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [03:43:09] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.500488/1.50, alarm hl:mem_free=814.000000M/100M [03:46:49] Load avg. on nightshade is WARNING: WARNING - load average: 15.73, 15.37, 13.65 [03:48:38] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.445 second response time [03:49:29] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 272 bytes in 9.167 second response time [03:52:39] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:52:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:57:09] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [04:02:10] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.682129/1.50, alarm hl:mem_free=819.000000M/100M [04:23:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:23:09] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:23:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:23:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256861 MB (4% inode=31%): [04:23:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68650 MB (7% inode=99%): [04:30:50] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.233399/2.00, alarm hl:mem_free=138.000000M/250M [04:32:51] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:47:51] Load avg. on nightshade is WARNING: WARNING - load average: 16.78, 15.73, 13.68 [04:49:50] Load avg. on nightshade is OK: OK - load average: 9.52, 13.82, 13.25 [04:53:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:02:50] Load avg. on nightshade is WARNING: WARNING - load average: 14.15, 17.49, 15.15 [05:23:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:23:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:23:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:23:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256689 MB (4% inode=31%): [05:24:00] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.545410/1.50, alarm hl:mem_free=545.000000M/100M [05:24:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68410 MB (7% inode=99%): [05:25:00] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:32:50] Load avg. on nightshade is WARNING: WARNING - load average: 20.06, 20.93, 17.87 [05:32:59] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.682129/1.50, alarm hl:mem_free=729.000000M/100M [05:38:19] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.174805/1.50, alarm hl:mem_free=779.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=2.157715/2.00, alarm hl:mem_free=779.000000M/250M [05:38:49] Load avg. on willow is WARNING: WARNING - load average: 21.27, 17.05, 13.95 [05:41:50] Load avg. on willow is OK: OK - load average: 9.42, 13.87, 13.27 [05:53:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:00:50] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 31.12, 24.50, 21.13 [06:02:59] Load avg. on willow is WARNING: WARNING - load average: 17.93, 18.24, 15.46 [06:23:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:23:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:23:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:23:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256711 MB (4% inode=31%): [06:24:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68494 MB (7% inode=99%): [06:30:59] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.03, 19.84, 17.42 [06:31:59] Load avg. on willow is WARNING: WARNING - load average: 26.88, 20.72, 17.90 [06:33:10] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=2.553223/1.50, alarm hl:mem_free=234.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=2.240723/2.00, alarm hl:mem_free=234.000000M/250M [06:38:39] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=3.512207/1.50, alarm hl:mem_free=643.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=3.697266/2.00, alarm hl:mem_free=643.000000M/250M [06:53:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.96, 22.59, 20.09 [07:00:59] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 16.44, 18.38, 22.60 [07:04:59] Load avg. on nightshade is WARNING: WARNING - load average: 7.02, 12.53, 19.15 [07:07:59] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 86823 MB (9% inode=99%): [07:21:59] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 33.52, 23.61, 19.66 [07:22:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:23:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:23:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:23:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:23:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256666 MB (4% inode=31%): [07:24:00] Load avg. on nightshade is WARNING: WARNING - load average: 20.42, 22.97, 19.99 [07:25:00] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 22.19, 23.19, 20.27 [07:25:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68296 MB (7% inode=99%): [07:29:59] Load avg. on willow is WARNING: WARNING - load average: 15.66, 17.69, 19.89 [07:33:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:33:20] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=2.109375/1.50, alarm hl:mem_free=357.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=2.401367/2.00, alarm hl:mem_free=357.000000M/250M [07:33:59] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 8.769 second response time [07:36:59] Load avg. on nightshade is WARNING: WARNING - load average: 11.66, 16.94, 19.50 [07:38:39] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.267090/1.50, alarm hl:mem_free=948.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=2.461426/2.00, alarm hl:mem_free=948.000000M/250M [07:45:59] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 44.65, 26.74, 22.27 [07:49:58] Load avg. on willow is OK: OK - load average: 10.43, 12.56, 14.69 [07:54:59] Load avg. on nightshade is WARNING: WARNING - load average: 9.92, 16.41, 19.45 [07:57:00] Load avg. on willow is WARNING: WARNING - load average: 18.09, 15.22, 14.98 [07:57:59] Load avg. on willow is OK: OK - load average: 13.37, 14.43, 14.72 [08:04:20] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:07:20] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.606445/1.50, alarm hl:mem_free=789.000000M/100M [08:08:39] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [08:09:00] Load avg. on nightshade is OK: OK - load average: 8.78, 11.00, 14.76 [08:09:20] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:12:39] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.685059/1.50, alarm hl:mem_free=972.000000M/100M [08:12:59] Load avg. on willow is WARNING: WARNING - load average: 16.50, 14.33, 13.85 [08:18:39] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [08:23:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:23:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:23:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:23:39] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:24:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256507 MB (4% inode=31%): [08:25:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68168 MB (6% inode=99%): [08:30:39] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.830078/1.50, alarm hl:mem_free=901.000000M/100M [09:03:23] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.502930/1.50, alarm hl:mem_free=566.000000M/100M [09:04:23] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:15:24] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.804688/1.50, alarm hl:mem_free=298.000000M/100M [09:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:24:04] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:24:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:24:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256444 MB (4% inode=31%): [09:24:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:26:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68048 MB (6% inode=99%): [09:32:05] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.757324/1.50, alarm hl:mem_free=1240.000000M/100M [09:36:14] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.231445/1.00, alarm hl:mem_free=16005.000000M/100M [09:38:02] Krinkle * Re: [Toolserver-l] PHP error reporting? [09:40:04] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [09:43:04] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.917480/1.50, alarm hl:mem_free=1096.000000M/100M [09:47:13] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [09:50:23] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.552734/1.50, alarm hl:mem_free=424.000000M/100M [09:52:23] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:59:23] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.626465/1.50, alarm hl:mem_free=626.000000M/100M [10:03:04] Load avg. on willow is WARNING: WARNING - load average: 15.16, 15.45, 13.36 [10:05:05] Load avg. on willow is OK: OK - load average: 12.99, 14.77, 13.37 [10:16:13] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 42.46, 24.65, 16.73 [10:17:13] Load avg. on nightshade is WARNING: WARNING - load average: 29.50, 24.24, 17.09 [10:18:14] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 39.08, 27.95, 18.87 [10:19:14] Load avg. on willow is WARNING: WARNING - load average: 17.86, 15.82, 14.36 [10:24:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:24:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:24:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256304 MB (4% inode=31%): [10:24:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:26:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67930 MB (6% inode=99%): [10:30:03] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.862305/1.50, alarm hl:mem_free=1021.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=2.160645/2.00, alarm hl:mem_free=1021.000000M/250M [10:38:04] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [10:48:03] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.528320/1.50, alarm hl:mem_free=1323.000000M/100M [10:49:03] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [10:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:59:34] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.568848/1.50, alarm hl:mem_free=578.000000M/100M [11:09:34] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [11:12:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.634277/1.50, alarm hl:mem_free=237.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.682617/2.00, alarm hl:mem_free=237.000000M/250M [11:19:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [11:24:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:24:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:24:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256186 MB (4% inode=31%): [11:24:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:26:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67818 MB (6% inode=99%): [11:46:15] Load avg. on nightshade is WARNING: WARNING - load average: 16.48, 13.77, 11.62 [11:47:13] Load avg. on nightshade is OK: OK - load average: 12.94, 13.29, 11.59 [11:48:03] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.645996/1.50, alarm hl:mem_free=924.000000M/100M [11:50:03] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [11:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:02:14] Load avg. on nightshade is WARNING: WARNING - load average: 17.95, 15.29, 12.98 [12:03:03] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.922363/1.50, alarm hl:mem_free=899.000000M/100M [12:04:44] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.320312/2.00, alarm hl:mem_free=187.000000M/250M [12:06:44] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:20:43] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.310547/2.00, alarm hl:mem_free=227.000000M/250M [12:24:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256199 MB (4% inode=31%): [12:24:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:24:25] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:24:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:26:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68631 MB (7% inode=99%): [12:46:33] Load avg. on nightshade is WARNING: WARNING - load average: 18.70, 12.86, 12.00 [12:46:33] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.129883/1.50, alarm hl:mem_free=979.000000M/100M [12:46:34] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 30.24, 19.40, 14.69 [12:46:34] Load avg. on nightshade is WARNING: WARNING - load average: 19.94, 18.40, 14.64 [12:52:13] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [12:52:53] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.167480/2.00, alarm hl:mem_free=227.000000M/250M [12:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:58:53] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:05:53] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.188476/2.00, alarm hl:mem_free=231.000000M/250M [13:10:28] 3(resolved) [ACC-239] Clicking "Quick Reply" sends the user to a blank page <10https://jira.toolserver.org/browse/ACC-239> (Manish Goregaokar) [13:24:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:25:24] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 255834 MB (4% inode=31%): [13:25:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:25:24] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:27:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68509 MB (7% inode=99%): [13:29:50] @replag [13:29:50] DaBPunkt: No replag found. Use "@replag all" to see all lags. [13:34:23] 3(assigned) [TS-1247] Create MMP merlbot <10https://jira.toolserver.org/browse/TS-1247> (DaB.) [13:34:24] 3(work started) [TS-1247] Create MMP merlbot <10https://jira.toolserver.org/browse/TS-1247> (DaB.) [13:38:21] 3(resolved) [TS-1247] Create MMP merlbot <10https://jira.toolserver.org/browse/TS-1247> (DaB.) [13:38:25] 3(work started) [TS-1246] MMP for the Look-and-Listen-Map <10https://jira.toolserver.org/browse/TS-1246> (DaB.) [13:46:13] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.703125/1.50, alarm hl:mem_free=836.000000M/100M [13:49:13] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [13:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:54:20] 3(work stopped) [TS-1246] MMP for the Look-and-Listen-Map <10https://jira.toolserver.org/browse/TS-1246> (DaB.) [13:57:22] 3(resolved) [TS-1246] MMP for the Look-and-Listen-Map <10https://jira.toolserver.org/browse/TS-1246> (DaB.) [13:58:01] DaBPunkt: if I have to do anything for the TS-1246, you can faster contact me here currently perhaps [14:01:16] ;) okay, I was a little bit too fast, got your mail [14:04:21] 3(work started) [TS-1245] Add wikis to database <10https://jira.toolserver.org/browse/TS-1245> (DaB.) [14:04:23] 3(resolved) [ACC-236] click on "Create" generates a new empty window/tab <10https://jira.toolserver.org/browse/ACC-236> (Manish Goregaokar) [14:08:22] DaBPunkt: contact! ;) What about postgresql-access? (and tomcat, but for my work postgres currently is more urgent to get the database schema) [14:09:02] wait... probably that's already done, didn't test yet... sorry [14:09:39] jongleur: I have no idea about the postgres-access, sorry. [14:09:50] apmon: How is the postgres-access handled at OSM? [14:10:10] does is use LDAP? [14:12:20] 3(commented) [ACC-237] empty string as username <10https://jira.toolserver.org/browse/ACC-237> (Manish Goregaokar) [14:12:21] 3(resolved) [TS-1245] Add wikis to database <10https://jira.toolserver.org/browse/TS-1245> (DaB.) [14:12:22] all I know about it yet is this: https://wiki.toolserver.org/view/OpenStreetMap and that's not working, yet. [14:12:29] Currently everyone should have read access to postgresql from willow [14:12:45] ah, okay, logged in via nightshade... [14:13:09] should be the same from nightshade (although I haven't tried it) [14:13:23] FATAL: role "jongleur"§ does not exist [14:13:31] how it is internally setup I don't know. You'll need to ask nosy about that [14:14:46] Does psql -h sql-mapnik osm_mapnik give you that error? [14:15:01] yes [14:15:31] (from my user account, not the project account, so that should not be a update problem, I think) [14:16:18] doesn't work from the mmp too. But does work from my normal user-account [14:17:11] DaBPunkt, thanks for adding those wikis. How long it will take to take effect? [14:18:08] * Tanvir confirms: It has not taken effect yet. [14:18:09] Tanvir: the views are already there. sql-toolserver.wiki was updated. The copies of toolserver.wikis will take an hour or so to update [14:18:54] Okay, thanks for the info. [14:19:20] 3(resolved) [TS-1131] Please add new ssh public key to my account <10https://jira.toolserver.org/browse/TS-1131> (DaB.) [14:19:38] apmon: so I have to contact nosy with regard to the postgres access? [14:20:00] jongleur: I will speak with nosy. I guess there is a bug somewhere [14:20:07] k, thanks [14:23:15] hopefully last question for now: what about the project database? Is there one already for lalm? [14:23:22] 3(work started) [TS-1242] Install tdom package <10https://jira.toolserver.org/browse/TS-1242> (DaB.) [14:23:23] 3(assigned) [TS-1242] Install tdom package <10https://jira.toolserver.org/browse/TS-1242> (DaB.) [14:23:40] looking at \du output from postgres, a lot of roles are defined. I am wondering if simply all users at the time were added manually? [14:23:59] I doubt that [14:24:15] apmon: can you add jongleur manualy? [14:24:25] (or tell me how to do it) [14:24:45] well, jongleur or lalm - as on the long term we need access from the project, right? [14:24:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:25:20] jongleur: both need to be added, but 1 step after the other :-) [14:25:26] of course ;) [14:25:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 255682 MB (4% inode=31%): [14:25:34] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:25:34] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:25:50] Only the roles gis, postgres and root have the privilege to add roles, so I don't think I can. [14:27:08] assuming the roles were manually added (rather than somehow automatically), the following command on ptolemy as should add a new role: [14:27:19] createuser jongleur [14:27:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68376 MB (7% inode=99%): [14:28:33] There is no createuser in my PATH, is it somewhere special? [14:30:10] one copy is in /usr/postgres/9.1-32bit/bin/ [14:30:31] You can probably use one of the multiple other copies as wel... [14:32:09] apmon: "Shall the new role be allowed to create databases?" [14:32:32] I don't think so [14:32:59] currently none of the roles (other than the super user roles) currently have any of those priviledges [14:33:04] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.219238/2.00, alarm hl:mem_free=220.000000M/250M [14:33:05] as long as anyone creates the database for that role, that's not necessary [14:33:31] I suspect you'll just manually create a new db u_jongleur with ownership to jongleur [14:33:52] or in this case I guess to the mmp user [14:34:24] apmon: how I do this? [14:34:55] There is a createdb command in that directory [14:35:02] -O will specify the owner of the db [14:40:03] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [14:43:18] ok, done [14:43:45] psql login works from the lalm account... [14:44:03] If you look at \du output, pretty much all users are in either the osm or osmuser role. I am not sure what that is about though [14:44:17] or who is in osm and who in osmuser [14:44:25] and works from jongleur, too. thanks [14:44:40] jongleur: both you and the mmp should have access and a db now [14:44:58] can you see if you can query the db, or if there are any other permissions issues? [14:45:16] well, tested only osm_mapnik yet, let's see [14:45:28] DaBPunkt: On a different note, do you have thoughts about the malloc issue? [14:45:29] the user database(s) are mysql? [14:45:34] Load avg. on nightshade is WARNING: WARNING - load average: 28.34, 16.36, 12.50 [14:46:12] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.023926/1.50, alarm hl:mem_free=757.000000M/100M [14:46:35] DaBPunkt also created a user database on postgres (db u_jongleur instead of osm_mapnik) [14:47:23] pardon? [14:48:59] apmon, DaBPunkt: thanks, works [14:49:12] Interesting, nosy actually set up the other db's to be owned by postgres rather than the user [14:49:26] psql -l gives a view of all the current db's available [14:49:34] Load avg. on nightshade is OK: OK - load average: 9.25, 13.72, 12.38 [14:49:48] psql -h sql-mapnik -l if you are not on ptolemy [14:50:48] great, thanks [14:51:36] okay... I will have to change my java code to deal with the mapnik scheme first, but now I can do that ;) I'll contact you again afterwards. Thanks [14:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:53:13] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [14:54:14] jongleur: Try not to kill the server with too complicated queries... ;-) It is kind or reeling under load anyway already. [14:54:24] I know [14:54:39] ;) for now I only need to query the database schema [14:55:17] It should survive that... :-) [14:55:22] but I know already about the load problems and I have an idea how to possibly circumvent the missing index on the tables I need (where adding an index is not an option..) [14:55:38] DaBPunkt: What was the pardon referring to? [14:56:02] " DaBPunkt also created a user database on postgres (db u_jongleur instead of osm_mapnik)" i eas not usre if that was a question for me [14:56:14] sure [14:56:39] I'm a little special case as I need the "raw" tables while the usual mapnik applications need the processed tables. For performance reasons there's no geographic index on the raw tables, but it should be possible to use the index on the processed ones probably [14:57:39] no, that was a statement :-) [15:03:13] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.706543/1.50, alarm hl:mem_free=885.000000M/100M [15:05:46] DaBPunkt: Do you think it is OK to try the dlmalloc with the mapnik rendering backends? I need to restart it to free up some memory again, anyway [15:06:17] sure. [15:24:55] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:25:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 255490 MB (4% inode=31%): [15:25:53] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:25:53] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:27:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68248 MB (7% inode=99%): [15:28:13] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.701660/1.50, alarm hl:mem_free=1011.000000M/100M [15:33:14] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [15:38:53] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.132812/1.00, alarm hl:mem_free=15642.000000M/100M [15:49:04] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [15:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:17:20] 3(resolved) [TS-1242] Install tdom package <10https://jira.toolserver.org/browse/TS-1242> (DaB.) [16:25:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:25:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 255389 MB (4% inode=31%): [16:26:04] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:26:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:27:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68097 MB (6% inode=99%): [16:33:23] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.515137/1.50, alarm hl:mem_free=1283.000000M/100M [16:34:23] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [16:38:18] in about 20 minutes we're holding a discussion about MediaWiki development switching from Subversion to git in #wikimedia-dev [16:38:31] There's going to be a second meeting in ~7 hours discussing largely the same topics, if that's better for you [16:38:49] live notes in http://etherpad.wikimedia.org/Git [16:39:12] please also add questions there if you have them [16:53:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:10:55] ok, MediaWiki git migration discussion happening now in #wikimedia-dev, another meeting happening in about 7 hours [17:25:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:25:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 255258 MB (4% inode=31%): [17:26:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:26:03] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:27:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67960 MB (6% inode=99%): [17:29:23] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.859375/1.50, alarm hl:mem_free=1261.000000M/100M [17:35:23] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [17:37:50] [[Special:Log/newusers]] create 10 * सरोज कुमार ढकाल * (New user account) [17:53:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:25:03] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:25:44] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 255034 MB (4% inode=31%): [18:26:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:26:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:27:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67813 MB (6% inode=99%): [18:27:44] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 40.27, 22.70, 13.93 [18:28:23] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.811523/1.50, alarm hl:mem_free=1148.000000M/100M [18:29:43] Load avg. on nightshade is WARNING: WARNING - load average: 23.86, 22.92, 15.14 [18:30:43] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 34.88, 26.06, 16.73 [18:37:23] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [18:39:03] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.107422/1.00, alarm hl:mem_free=15717.000000M/100M [18:48:04] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [19:08:43] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 86302 MB (9% inode=99%): [19:12:29] anybody here who is able to discuss some osm_mapnik-database issues? [19:23:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:25:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:25:17] toolserver: 03dab * r1122 10/trunk/ts-specs/tcl85/TStcl85-tdom.spec: Added TStcl85-tdom.spec to fix TS-1242. [19:26:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:26:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:26:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 254849 MB (4% inode=31%): [19:27:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67626 MB (6% inode=99%): [19:48:58] Hey there, does anyone here know whether Toolserver has any sort of agreement with Freenode in any way ? [19:49:21] Like number of users per IP or message buffer / second? [20:23:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:25:04] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:26:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:26:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:26:44] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 254594 MB (4% inode=31%): [20:27:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68410 MB (7% inode=99%): [20:35:14] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.750488/2.00, alarm hl:mem_free=179.000000M/250M [20:36:13] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [21:00:20] 3(reopened) [RIVER-22] Request for installation of Babel python module <10https://jira.toolserver.org/browse/RIVER-22> (Snottywong) [21:06:21] 3(commented) [RIVER-22] Request for installation of Babel python module <10https://jira.toolserver.org/browse/RIVER-22> (Marlen Caemmerer) [21:07:33] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.558105/1.50, alarm hl:mem_free=621.000000M/100M [21:13:33] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [21:16:33] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.783203/1.50, alarm hl:mem_free=909.000000M/100M [21:22:43] Load avg. on nightshade is WARNING: WARNING - load average: 17.50, 15.81, 13.25 [21:23:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:24:44] Load avg. on nightshade is OK: OK - load average: 13.00, 14.94, 13.25 [21:26:03] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:26:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:26:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:26:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 254449 MB (4% inode=31%): [21:27:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68232 MB (7% inode=99%): [21:27:43] Load avg. on nightshade is WARNING: WARNING - load average: 19.96, 17.52, 14.62 [21:39:13] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.131836/1.00, alarm hl:mem_free=15683.000000M/100M [21:49:13] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [21:51:21] 3(commented) [RIVER-22] Request for installation of Babel python module <10https://jira.toolserver.org/browse/RIVER-22> (Snottywong) [21:55:43] Load avg. on nightshade is WARNING: WARNING - load average: 18.95, 15.43, 14.27 [21:59:44] Load avg. on nightshade is OK: OK - load average: 12.57, 14.68, 14.28 [22:12:45] Load avg. on nightshade is WARNING: WARNING - load average: 15.86, 14.81, 14.24 [22:26:13] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:26:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:26:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:26:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 254441 MB (4% inode=31%): [22:27:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68053 MB (6% inode=99%): [22:32:33] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.628418/1.50, alarm hl:mem_free=421.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=2.368652/2.00, alarm hl:mem_free=421.000000M/250M [22:36:54] Load avg. on nightshade is WARNING: WARNING - load average: 20.18, 20.39, 19.25 [22:49:33] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [22:49:53] Load avg. on nightshade is OK: OK - load average: 6.38, 10.34, 14.62 [22:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:10:37] nacht ts [23:26:14] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:26:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:26:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:26:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 252713 MB (4% inode=30%): [23:27:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67862 MB (6% inode=99%): [23:53:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out)