[00:02:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.969238/1.75, alarm hl:np_load_avg=0.763672/2.00, alarm hl:mem_free=287.000000M/300M [00:03:23] 3(assigned) [ACCAPP-443] Wikimedia Hackathon <10https://jira.toolserver.org/browse/ACCAPP-443> (Daniel Kinzler) [00:03:31] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:04:36] [[Admin:Mkuser]] N 10https://wiki.toolserver.org/w/index.php?oldid=6561&rcid=8631 * Daniel * (+30) (#REDIRECT [[Admin:Add_a_user]]) [00:06:52] [[Admin:Add a user]] 10https://wiki.toolserver.org/w/index.php?diff=6562&oldid=3576&rcid=8632 * Daniel * (+9) (as root!) [00:10:31] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9622.000000 [00:11:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [00:15:06] [[Admin:Add a user]] 10https://wiki.toolserver.org/w/index.php?diff=6563&oldid=6562&rcid=8633 * Daniel * (-8) (mkuser is on willow now, apparently) [00:17:31] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [00:17:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:19:12] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [00:20:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:23:21] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:26:20] 3(commented) [ACCAPP-442] Operate a bot and editor tools <10https://jira.toolserver.org/browse/ACCAPP-442> (Daniel Kinzler) [00:35:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53148 MB (5% inode=99%): [00:46:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:10:31] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10295.000000 [01:11:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6549.000000 [01:17:41] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [01:18:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:19:22] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [01:20:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:23:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:36:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53015 MB (5% inode=99%): [01:46:21] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:10:32] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12359.000000 [02:11:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9989.000000 [02:17:41] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [02:18:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:19:21] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [02:21:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:24:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:36:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52879 MB (5% inode=99%): [02:47:21] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:07:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.443359/1.75, alarm hl:np_load_avg=0.454590/2.00, alarm hl:mem_free=270.000000M/300M [03:11:31] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14759.000000 [03:12:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12623.000000 [03:13:32] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:17:41] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [03:19:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:20:21] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [03:21:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:24:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:37:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52749 MB (5% inode=99%): [03:47:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:11:32] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15548.000000 [04:13:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11867.000000 [04:17:41] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [04:20:23] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [04:20:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:21:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:25:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:26:01] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2535.000000 [04:27:01] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1430.000000 [04:38:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52603 MB (5% inode=99%): [04:48:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:10:41] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3586.000000 [05:14:41] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1400.000000 [05:17:42] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [05:20:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:21:23] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [05:21:41] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:25:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:39:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52337 MB (5% inode=99%): [05:48:31] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:18:41] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [06:20:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:21:32] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [06:21:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:25:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:39:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52349 MB (5% inode=99%): [06:44:51] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.162109/1.00, alarm hl:np_load_long=0.679688/1.50, alarm hl:mem_free=24287.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.162109/1.10, alarm hl:np_load_long=0.679688/1.75, alarm hl:mem_free=24287.000000M/300M [06:45:54] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [06:48:32] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:18:53] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [07:20:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:21:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [07:21:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:25:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:39:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53172 MB (5% inode=99%): [07:49:31] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:05:42] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1941.000000 [08:08:42] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1692.000000 [08:09:02] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 38853 MB (4% inode=99%): [08:19:52] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [08:20:53] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:21:41] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [08:21:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:26:41] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:39:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53041 MB (5% inode=99%): [08:49:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:20:52] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [09:21:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [09:21:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:22:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:27:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:39:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52938 MB (5% inode=99%): [09:50:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:20:52] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [10:21:43] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [10:22:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:22:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:27:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:40:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52816 MB (5% inode=99%): [10:50:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:21:51] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [11:22:41] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [11:22:51] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:22:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:27:41] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:40:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52671 MB (5% inode=99%): [11:50:42] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:22:01] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [12:22:42] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [12:23:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:23:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:27:41] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:41:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52513 MB (5% inode=99%): [12:51:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:22:01] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [13:22:43] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [13:23:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:23:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:27:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:41:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52375 MB (5% inode=99%): [13:52:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:22:02] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [14:23:43] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [14:24:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:24:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:28:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:41:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52258 MB (5% inode=99%): [14:53:01] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:12:51] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 12.904 second response time [15:13:01] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.494141/1.75, alarm hl:np_load_avg=0.446289/2.00, alarm hl:mem_free=222.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.494141/1.50, alarm hl:np_load_long=0.399902/1.75, alarm hl:mem_free=222.000000M/250M [15:14:00] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:22:02] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [15:23:43] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [15:25:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:25:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:29:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:41:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52995 MB (5% inode=99%): [16:17:32] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1881.000000 [16:18:11] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1901 [16:23:01] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [16:23:52] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [16:25:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:25:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:29:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:33:32] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1781.000000 [16:34:11] MySQL slave on thyme is OK: Uptime: 1339422 Threads: 19 Questions: 415312668 Slow queries: 216177 Opens: 73140 Flush tables: 1 Open tables: 2608 Queries per second avg: 310.68 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1724 [16:41:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52845 MB (5% inode=99%): [17:23:00] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [17:23:52] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [17:25:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:25:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:26:01] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.816 second response time [17:29:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:41:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52684 MB (5% inode=99%): [18:02:16] Load avg. on amaranth is CRITICAL: CRITICAL - load average: 33.59, 20.82, 12.61 [18:03:23] Load avg. on amaranth is WARNING: WARNING - load average: 18.66, 19.14, 12.61 [18:06:22] Load avg. on amaranth is OK: OK - load average: 8.12, 14.11, 11.78 [18:23:23] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [18:23:51] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [18:25:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:25:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:26:01] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:29:21] 3(created) [MNT-1175] Add and correct new mgmt-IPs.; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1175> (DaB.) [18:29:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:37:20] 3(resolved) [MNT-1175] Add and correct new mgmt-IPs. <10https://jira.toolserver.org/browse/MNT-1175> (DaB.) [18:41:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52491 MB (5% inode=99%): [19:15:37] [[Category:Communication]] ! 10https://wiki.toolserver.org/w/index.php?diff=6564&oldid=541&rcid=8634 * 110.39.170.216 * (+1809) () [19:16:44] [[Category:Communication]] ! 10https://wiki.toolserver.org/w/index.php?diff=6565&oldid=6564&rcid=8635 * 110.39.170.216 * (-12) () [19:21:47] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:110.39.170.21610]] with an expiry time of infinite (anonymous users only, account creation disabled): Spam/gibberish) [19:22:41] [[Category:Communication]] 10https://wiki.toolserver.org/w/index.php?diff=6566&oldid=6565&rcid=8637 * Valhallasw * (-1797) (Revert gibberish) [19:23:23] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [19:23:52] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [19:25:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:25:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:26:01] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:29:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:41:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52322 MB (5% inode=99%): [20:02:53] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.660156/1.50, alarm hl:np_load_long=0.967285/1.75, alarm hl:mem_free=3059.000000M/250M [20:03:52] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [20:09:24] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 37329 MB (3% inode=99%): [20:23:22] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [20:23:52] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [20:26:01] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:26:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:26:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:30:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:42:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52159 MB (5% inode=99%): [21:23:22] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [21:23:53] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [21:26:11] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:26:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:27:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:30:01] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 9.240 second response time [21:30:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:32:22] 3(created) [MNT-1176] Restarted fisheye on amaranth; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1176> (DaB.) [21:34:20] 3(updated) [MNT-1176] Restarted fisheye on amaranth <10https://jira.toolserver.org/browse/MNT-1176> (DaB.) [21:41:28] zzz =_= [21:43:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52905 MB (5% inode=99%): [22:23:53] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [22:24:22] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [22:27:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:27:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:30:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:44:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52738 MB (5% inode=99%): [22:59:01] nacht ts [23:24:01] ethernet 0/1/24 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:UP: 1 int NOK : CRITICAL [23:24:32] ethernet 0/1/18 on asw-oe16-esams.mgmt is CRITICAL: GigabitEthernet0/1/18:UP: 1 int NOK : CRITICAL [23:27:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:27:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:31:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:45:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52581 MB (5% inode=99%):