[00:01:28] Load avg. on willow is WARNING: WARNING - load average: 17.23, 14.78, 11.46 [00:01:29] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.875488/1.50, alarm hl:mem_free=535.000000M/100M [00:03:29] Load avg. on willow is OK: OK - load average: 12.79, 14.20, 11.66 [00:07:18] nacht ts [00:07:30] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:22:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:28:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 218046 MB (4% inode=27%): [00:28:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:29:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:30:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67520 MB (6% inode=99%): [00:33:29] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.101562/2.00, alarm hl:mem_free=232.000000M/250M [00:34:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:34:29] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:56:22] 3(created) [ACCAPP-428] Data Analyst at WMF requesting a new account; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-428> (Ayush Khanna) [01:22:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:27:57] zzz [01:28:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 219174 MB (4% inode=27%): [01:28:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:29:39] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:30:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67390 MB (6% inode=99%): [01:34:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:34:39] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.013672/2.00, alarm hl:mem_free=171.000000M/250M [01:37:50] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:56:50] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.831543/2.00, alarm hl:mem_free=173.000000M/250M [01:57:50] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:12:50] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.816406/2.00, alarm hl:mem_free=170.000000M/250M [02:22:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:28:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 219116 MB (4% inode=27%): [02:28:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:29:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:30:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67257 MB (6% inode=99%): [02:34:40] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:23:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:28:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 217799 MB (4% inode=27%): [03:28:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:29:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:30:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67109 MB (6% inode=99%): [03:33:00] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.184570/1.00, alarm hl:mem_free=15747.000000M/100M [03:35:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:42:10] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.789062/2.00, alarm hl:mem_free=231.000000M/250M [03:43:09] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:58:09] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [04:22:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:28:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 217733 MB (4% inode=27%): [04:28:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:29:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:30:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66975 MB (6% inode=99%): [04:30:15] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6456&oldid=6443&rcid=8474 * 91.198.174.202 * (+168) (updated page) [04:35:38] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:22:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:28:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221581 MB (4% inode=28%): [05:28:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:30:09] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:30:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66707 MB (6% inode=99%): [05:35:09] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.015625/1.00, alarm hl:mem_free=15461.000000M/100M [05:35:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:36:08] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [05:46:09] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.026367/1.00, alarm hl:mem_free=15326.000000M/100M [06:07:20] 3(created) [MNT-1162] Give read permissions on the way/node/rels tables; Maintenance: ptolemy; Minor Minor work <10https://jira.toolserver.org/browse/MNT-1162> (Kai Krueger) [06:11:09] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.501465/1.50, alarm hl:mem_free=432.000000M/100M [06:13:09] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.283203/1.00, alarm hl:mem_free=15020.000000M/100M [06:14:10] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:17:09] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.357422/2.00, alarm hl:mem_free=217.000000M/250M [06:19:09] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [06:22:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:23:09] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.046875/1.00, alarm hl:mem_free=15058.000000M/100M [06:26:25] 3(resolved) [MNT-1162] Give read permissions on the way/node/rels tables <10https://jira.toolserver.org/browse/MNT-1162> (Kai Krueger) [06:26:26] 3(commented) [TS-1246] MMP for the Look-and-Listen-Map <10https://jira.toolserver.org/browse/TS-1246> (Kai Krueger) [06:28:09] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [06:28:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221499 MB (4% inode=28%): [06:29:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:30:09] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:30:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66742 MB (6% inode=99%): [06:31:49] Load avg. on willow is WARNING: WARNING - load average: 19.30, 17.20, 13.68 [06:33:49] Load avg. on willow is OK: OK - load average: 11.63, 14.81, 13.20 [06:35:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:45:19] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.156250/1.00, alarm hl:mem_free=15023.000000M/100M [06:46:50] Load avg. on willow is WARNING: WARNING - load average: 19.68, 15.51, 13.49 [06:47:19] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [06:55:19] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.128906/1.00, alarm hl:mem_free=15054.000000M/100M [07:01:19] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.815918/1.50, alarm hl:mem_free=214.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.744629/2.00, alarm hl:mem_free=214.000000M/250M [07:05:19] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:10:29] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 83625 MB (8% inode=99%): [07:22:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:29:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221320 MB (4% inode=28%): [07:29:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:30:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:30:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66625 MB (6% inode=99%): [07:36:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:48:00] Load avg. on nightshade is WARNING: WARNING - load average: 16.77, 13.73, 9.70 [07:48:19] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.721680/1.50, alarm hl:mem_free=1318.000000M/100M [07:49:00] Load avg. on nightshade is OK: OK - load average: 12.13, 13.06, 9.73 [07:50:19] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [07:55:19] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.207031/1.00, alarm hl:mem_free=15230.000000M/100M [08:00:19] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [08:20:20] 3(reopened) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (Roan Kattouw) [08:23:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:29:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221175 MB (4% inode=28%): [08:29:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:30:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:30:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67454 MB (6% inode=99%): [08:37:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:52:28] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.375000/1.00, alarm hl:mem_free=14912.000000M/100M [08:57:29] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [09:00:29] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.144531/1.00, alarm hl:mem_free=15092.000000M/100M [09:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:29:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221034 MB (4% inode=28%): [09:29:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:30:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:30:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67342 MB (6% inode=99%): [09:37:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:01:22] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.112305/1.00, alarm hl:mem_free=14863.000000M/100M [10:23:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:29:22] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220952 MB (4% inode=28%): [10:30:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:30:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:30:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67218 MB (6% inode=99%): [10:37:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:01:23] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.034180/1.00, alarm hl:mem_free=14111.000000M/100M [11:16:32] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [11:23:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:27:32] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.105469/1.00, alarm hl:mem_free=14306.000000M/100M [11:29:21] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220549 MB (4% inode=28%): [11:30:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:30:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:30:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67093 MB (6% inode=99%): [11:36:32] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [11:37:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:17:32] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.081055/1.00, alarm hl:mem_free=14381.000000M/100M [12:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:29:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220400 MB (4% inode=28%): [12:30:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:30:53] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:30:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66980 MB (6% inode=99%): [12:37:21] 3(resolved) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (DaB.) [12:37:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:53:43] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [13:03:44] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.008789/1.00, alarm hl:mem_free=14324.000000M/100M [13:08:43] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [13:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:29:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220390 MB (4% inode=28%): [13:30:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66850 MB (6% inode=99%): [13:30:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:30:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:35:43] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.322266/1.00, alarm hl:mem_free=14090.000000M/100M [13:37:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:07:22] 3(commented) [ACCAPP-417] Toolserver access request <10https://jira.toolserver.org/browse/ACCAPP-417> (Sylvain Boissel) [14:20:32] hello all [14:23:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:23:35] @replag [14:23:35] DaBPunkt: No replag found. Use "@replag all" to see all lags. [14:23:38] :) [14:29:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220010 MB (4% inode=28%): [14:31:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:31:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:31:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66726 MB (6% inode=99%): [14:35:53] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.264649/1.00, alarm hl:mem_free=15308.000000M/100M [14:37:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:04:07] [[Special:Log/newusers]] create 10 * Tataguez * (New user account) [15:13:52] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.959961/2.00, alarm hl:mem_free=198.000000M/250M [15:15:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:29:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 218212 MB (4% inode=27%): [15:31:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:31:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:31:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66537 MB (6% inode=99%): [15:36:53] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.327149/1.00, alarm hl:mem_free=15190.000000M/100M [15:37:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:07:20] 3(commented) [ACCAPP-413] I want to operate my global bot "GedawyBot" on toolserver in order to add and update interwikis <10https://jira.toolserver.org/browse/ACCAPP-413> (DaB.) [16:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:29:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 214661 MB (4% inode=27%): [16:31:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:31:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:32:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67326 MB (6% inode=99%): [16:34:21] 3(updated) [ACCAPP-421] Export tools for Wikisource <10https://jira.toolserver.org/browse/ACCAPP-421> (Thomas Pellissier Tanon) [16:36:22] 3(commented) [ACCAPP-421] Export tools for Wikisource <10https://jira.toolserver.org/browse/ACCAPP-421> (Thomas Pellissier Tanon) [16:36:52] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.112305/1.00, alarm hl:mem_free=15093.000000M/100M [16:38:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:56:21] 3(updated) [ACCAPP-426] TjBot's request on ToolServer for Interwiki from id.wikipedia <10https://jira.toolserver.org/browse/ACCAPP-426> (Moel) [17:03:41] 3(updated) [ACCAPP-423] Studying the relationship between box office revenues of Hollywood films and the statistics on their Wikipedia articles <10https://jira.toolserver.org/browse/ACCAPP-423> [17:12:52] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.965332/2.00, alarm hl:mem_free=208.000000M/250M [17:13:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [17:16:21] 3(commented) [ACCAPP-423] Studying the relationship between box office revenues of Hollywood films and the statistics on their Wikipedia articles <10https://jira.toolserver.org/browse/ACCAPP-423> [17:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:29:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 211685 MB (3% inode=27%): [17:31:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:31:22] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:32:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67158 MB (6% inode=99%): [17:36:22] 3(created) [ACCAPP-430] Bot - Sharing lists; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-430> (Marco) [17:37:01] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.207031/1.00, alarm hl:mem_free=14650.000000M/100M [17:38:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:13:02] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.997559/2.00, alarm hl:mem_free=206.000000M/250M [18:14:02] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [18:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:29:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210530 MB (3% inode=27%): [18:31:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:31:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:32:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67029 MB (6% inode=99%): [18:37:02] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.208984/1.00, alarm hl:mem_free=14841.000000M/100M [18:38:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:10:43] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 84112 MB (8% inode=99%): [19:14:42] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.051758/2.00, alarm hl:mem_free=211.000000M/250M [19:16:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [19:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:29:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 210023 MB (3% inode=27%): [19:31:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:31:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:32:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66887 MB (6% inode=99%): [19:37:42] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.182617/1.00, alarm hl:mem_free=14780.000000M/100M [19:38:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:45:42] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [20:03:53] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1852 [20:10:43] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.035156/1.00, alarm hl:mem_free=14601.000000M/100M [20:13:43] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [20:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:29:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209581 MB (3% inode=27%): [20:30:42] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.051758/1.00, alarm hl:mem_free=14697.000000M/100M [20:31:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:32:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:32:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66737 MB (6% inode=99%): [20:38:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:40:42] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1895 [20:40:42] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1898.000000 [20:42:42] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [20:54:03] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3601 [20:55:03] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3574 [21:02:43] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.015625/1.00, alarm hl:mem_free=14641.000000M/100M [21:23:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:24:43] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [21:27:42] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.020508/1.00, alarm hl:mem_free=14824.000000M/100M [21:29:42] MySQL slave on rosemary is OK: Uptime: 3641037 Threads: 43 Questions: 1057151281 Slow queries: 432826 Opens: 5517 Flush tables: 1 Open tables: 836 Queries per second avg: 290.343 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [21:29:43] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [21:30:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209163 MB (3% inode=27%): [21:32:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:32:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:32:42] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1898 [21:32:42] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66583 MB (6% inode=99%): [21:32:52] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1904.000000 [21:38:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:53:02] MySQL slave on z-dat-s3-a is OK: Uptime: 890010 Threads: 20 Questions: 859724760 Slow queries: 122883 Opens: 12878323 Flush tables: 2 Open tables: 16371 Queries per second avg: 965.972 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1691 [22:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:23:53] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [22:30:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 208990 MB (3% inode=26%): [22:32:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:32:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:32:52] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2401.000000 [22:33:42] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2435 [22:33:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66411 MB (6% inode=99%): [22:38:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:53:02] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.037109/1.00, alarm hl:mem_free=14878.000000M/100M [22:53:52] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1938 [22:54:02] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [23:07:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [23:07:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [23:10:15] [[User talk:Snowolf]] ! 10https://wiki.toolserver.org/w/index.php?diff=6457&oldid=4650&rcid=8476 * Snowolf * (-1370) (blanking) [23:10:35] [[User:Snowolf]] ! 10https://wiki.toolserver.org/w/index.php?diff=6458&oldid=4651&rcid=8477 * Snowolf * (-1372) (-obsolete notice) [23:11:42] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 15.339 second response time [23:12:25] 3(commented) [TS-1240] Make webserver logs easily accessible from login servers <10https://jira.toolserver.org/browse/TS-1240> (Tim.Landscheidt) [23:13:02] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.018555/1.00, alarm hl:mem_free=14898.000000M/100M [23:19:41] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:23:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:30:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 207183 MB (3% inode=26%): [23:30:52] /sql on adenia is WARNING: DISK WARNING - free space: /sql 66773 MB (10% inode=99%): [23:32:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:32:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66239 MB (6% inode=99%): [23:35:04] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [23:38:51] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 16.951 second response time [23:39:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:53:52] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2803 [23:56:52] /sql on adenia is CRITICAL: DISK CRITICAL - free space: /sql 37981 MB (5% inode=99%):