[00:07:22] 3(created) [TS-1284] chown; Toolserver; Trivial Task <10https://jira.toolserver.org/browse/TS-1284> (Andrew Pearson) [00:07:50] nacht ts [00:11:20] 3(commented) [TS-1284] chown <10https://jira.toolserver.org/browse/TS-1284> (DaB.) [00:15:21] 3(commented) [TS-1284] chown <10https://jira.toolserver.org/browse/TS-1284> (Andrew Pearson) [00:31:33] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.1% (339744 kB) free! [00:32:43] [[Special:Log/newusers]] create 10 * NorKo * (New user account) [00:38:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:38:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:38:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43156 MB (4% inode=99%): [00:41:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:41:52] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [00:42:33] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [00:42:33] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.9% (204440 kB) free! [01:17:33] Free Memory on damiana.esi is WARNING: WARNING - 10.5% (440384 kB) free! [01:19:33] Free Memory on damiana.esi is CRITICAL: CRITICAL - 9.7% (404996 kB) free! [01:26:32] Free Memory on damiana.esi is WARNING: WARNING - 10.3% (431672 kB) free! [01:27:20] 3(created) [ACCAPP-451] Urgent Account Creation; Account Approval; Critical New Account <10https://jira.toolserver.org/browse/ACCAPP-451> (Cyberpower678) [01:31:32] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.4% (353468 kB) free! [01:38:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:38:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:38:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43014 MB (4% inode=99%): [01:41:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:42:02] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [01:42:33] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [01:46:32] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.7% (194908 kB) free! [01:48:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.526367/1.75, alarm hl:np_load_avg=0.472656/2.00, alarm hl:mem_free=233.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.526367/1.50, alarm hl:np_load_long=0.478027/1.75, alarm hl:mem_free=233.000000M/250M [01:52:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:55:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.556641/1.75, alarm hl:np_load_avg=0.522461/2.00, alarm hl:mem_free=245.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.556641/1.50, alarm hl:np_load_long=0.504394/1.75, alarm hl:mem_free=245.000000M/250M [02:05:21] 3(created) [TS-1285] Retrieve data from expired user account; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1285> (Soxred93) [02:07:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:15:15] nosy: you around? [02:31:33] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.5% (353828 kB) free! [02:38:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:38:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:38:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42823 MB (4% inode=99%): [02:41:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:42:03] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [02:42:33] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [02:46:42] Free Memory on damiana.esi is CRITICAL: CRITICAL - 6.3% (261656 kB) free! [03:00:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.706055/1.75, alarm hl:np_load_avg=0.770019/2.00, alarm hl:mem_free=226.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.706055/1.50, alarm hl:np_load_long=0.613769/1.75, alarm hl:mem_free=226.000000M/250M [03:10:32] Free Memory on turnera.esi is WARNING: WARNING - 13.4% (562780 kB) free! [03:10:42] Free Memory on damiana.esi is WARNING: WARNING - 11.1% (464344 kB) free! [03:11:43] Free Memory on damiana.esi is CRITICAL: CRITICAL - 9.9% (416036 kB) free! [03:13:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:16:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.622559/1.75, alarm hl:np_load_avg=0.756836/2.00, alarm hl:mem_free=258.000000M/300M [03:28:20] 3(commented) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Andrew Pearson) [03:30:21] 3(commented) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Soxred93) [03:38:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:38:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:38:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43567 MB (4% inode=99%): [03:41:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:42:03] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [03:42:33] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [03:58:32] Free Memory on turnera.esi is CRITICAL: CRITICAL - 9.3% (387336 kB) free! [04:05:33] Free Memory on turnera.esi is WARNING: WARNING - 11.0% (458952 kB) free! [04:11:43] Free Memory on damiana.esi is CRITICAL: CRITICAL - 6.3% (264176 kB) free! [04:17:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.429199/1.75, alarm hl:np_load_avg=0.466309/2.00, alarm hl:mem_free=218.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.429199/1.50, alarm hl:np_load_long=0.458984/1.75, alarm hl:mem_free=218.000000M/250M [04:18:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:38:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:38:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:38:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43438 MB (4% inode=99%): [04:41:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:42:03] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [04:42:42] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [04:58:43] Free Memory on turnera.esi is CRITICAL: CRITICAL - 7.3% (304528 kB) free! [05:06:42] Free Memory on turnera.esi is WARNING: WARNING - 10.6% (442988 kB) free! [05:11:43] Free Memory on damiana.esi is CRITICAL: CRITICAL - 1.6% (65224 kB) free! [05:19:43] Free Memory on damiana.esi is WARNING: WARNING - 11.1% (465620 kB) free! [05:20:43] Free Memory on damiana.esi is CRITICAL: CRITICAL - 9.9% (415440 kB) free! [05:38:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:38:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:38:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43159 MB (4% inode=99%): [05:41:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:42:03] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [05:42:42] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [05:50:43] Free Memory on damiana.esi is WARNING: WARNING - 10.8% (452840 kB) free! [05:50:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.784668/1.75, alarm hl:np_load_avg=0.756836/2.00, alarm hl:mem_free=224.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.784668/1.50, alarm hl:np_load_long=0.681152/1.75, alarm hl:mem_free=224.000000M/250M [05:52:42] Free Memory on damiana.esi is CRITICAL: CRITICAL - 8.3% (346576 kB) free! [05:54:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:56:12] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.00, alarm hl:np_load_long=0.905273/1.50, alarm hl:mem_free=22721.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.10, alarm hl:np_load_long=0.905273/1.75, alarm hl:mem_free=22721.000000M/300M [05:57:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:58:43] Free Memory on turnera.esi is CRITICAL: CRITICAL - 6.6% (276492 kB) free! [06:05:43] Free Memory on turnera.esi is WARNING: WARNING - 10.3% (432692 kB) free! [06:07:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.847656/1.75, alarm hl:np_load_avg=0.812012/2.00, alarm hl:mem_free=250.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.847656/1.50, alarm hl:np_load_long=0.752930/1.75, alarm hl:mem_free=250.000000M/250M [06:13:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:30:42] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.187500/1.75, alarm hl:np_load_avg=0.914551/2.00, alarm hl:mem_free=213.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.187500/1.50, alarm hl:np_load_long=0.838379/1.75, alarm hl:mem_free=213.000000M/250M [06:35:00] zzz =__= [06:36:43] Free Memory on turnera.esi is CRITICAL: CRITICAL - 9.9% (415828 kB) free! [06:38:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:38:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:38:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43260 MB (4% inode=99%): [06:41:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:42:49] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:43:00] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [06:43:40] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [06:53:40] Free Memory on damiana.esi is CRITICAL: CRITICAL - 6.1% (254916 kB) free! [07:36:48] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.9% (374264 kB) free! [07:38:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:38:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:38:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43141 MB (4% inode=99%): [07:42:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:51:07] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [07:51:07] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [07:53:48] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.7% (197708 kB) free! [07:54:49] [[Special:Log/newusers]] create 10 * Secret of success * (New user account) [08:06:36] ignore *!~tsnag@* [08:06:44] I hate irc [08:11:52] [[Special:Log/newusers]] create 10 * Jafarpulpally * (New user account) [08:21:47] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 18014 MB (1% inode=99%): [08:35:59] [[Special:Log/newusers]] create 10 * Sarvaturi * (New user account) [08:36:58] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.3% (345540 kB) free! [08:38:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:38:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:38:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43014 MB (4% inode=99%): [08:42:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:43:58] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [08:44:37] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [08:53:48] Free Memory on damiana.esi is CRITICAL: CRITICAL - 7.8% (327528 kB) free! [09:10:08] [[User talk:Sarvaturi]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6652&rcid=8775 * Sarvaturi * (+1) (Created page with ".") [09:36:58] Free Memory on turnera.esi is CRITICAL: CRITICAL - 7.4% (311088 kB) free! [09:38:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:39:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42887 MB (4% inode=99%): [09:39:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:42:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:44:08] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [09:44:37] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [09:53:48] Free Memory on damiana.esi is CRITICAL: CRITICAL - 2.6% (107900 kB) free! [10:23:43] [[Admin:Datacenter-Tasks]] 10https://wiki.toolserver.org/w/index.php?diff=6653&oldid=6639&rcid=8776 * Nosy * (+115) () [10:36:57] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.7% (366140 kB) free! [10:38:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:39:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42763 MB (4% inode=99%): [10:39:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:42:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:44:08] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [10:44:37] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [10:53:48] Free Memory on damiana.esi is CRITICAL: CRITICAL - 7.3% (306972 kB) free! [11:36:48] Free Memory on damiana.esi is WARNING: WARNING - 16.5% (689584 kB) free! [11:36:58] Free Memory on turnera.esi is CRITICAL: CRITICAL - 7.5% (312008 kB) free! [11:38:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:39:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42645 MB (4% inode=99%): [11:39:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:42:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:42:47] Free Memory on damiana.esi is CRITICAL: CRITICAL - 9.1% (382168 kB) free! [11:44:08] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [11:44:48] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [12:30:40] [[Special:Log/newusers]] create 10 * Shuang42 * (New user account) [12:36:59] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.4% (351848 kB) free! [12:38:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:39:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43447 MB (4% inode=99%): [12:39:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:42:37] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:42:47] Free Memory on damiana.esi is CRITICAL: CRITICAL - 6.1% (253692 kB) free! [12:44:08] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [12:44:47] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [13:36:58] Free Memory on turnera.esi is CRITICAL: CRITICAL - 7.2% (301772 kB) free! [13:38:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:39:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43316 MB (4% inode=99%): [13:39:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:42:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:42:47] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.2% (174996 kB) free! [13:44:07] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [13:44:48] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [14:08:08] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.402832/1.75, alarm hl:np_load_avg=0.409668/2.00, alarm hl:mem_free=269.000000M/300M [14:14:08] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [14:26:38] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1955.000000 [14:36:58] Free Memory on turnera.esi is CRITICAL: CRITICAL - 8.3% (346016 kB) free! [14:38:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:39:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43123 MB (4% inode=99%): [14:39:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:42:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:42:48] Free Memory on damiana.esi is CRITICAL: CRITICAL - 2.1% (86772 kB) free! [14:44:08] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [14:44:48] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [14:58:38] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [15:13:12] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.560059/1.75, alarm hl:np_load_avg=0.528320/2.00, alarm hl:mem_free=207.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.560059/1.50, alarm hl:np_load_long=0.473633/1.75, alarm hl:mem_free=207.000000M/250M [15:16:11] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:33:11] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.402832/1.75, alarm hl:np_load_avg=0.419922/2.00, alarm hl:mem_free=269.000000M/300M [15:37:02] Free Memory on turnera.esi is CRITICAL: CRITICAL - 6.6% (277832 kB) free! [15:39:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:39:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42962 MB (4% inode=99%): [15:39:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:42:52] Free Memory on damiana.esi is CRITICAL: CRITICAL - 1.5% (63936 kB) free! [15:42:53] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:43:20] [[Special:Log/newusers]] create 10 * Yamaaan * (New user account) [15:44:12] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [15:44:52] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [15:58:52] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6299.000000 [16:37:12] Free Memory on turnera.esi is CRITICAL: CRITICAL - 6.8% (282776 kB) free! [16:39:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:39:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42770 MB (4% inode=99%): [16:39:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:43:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default svc:/application/management/common-agent-container-2:default [16:43:52] Free Memory on damiana.esi is CRITICAL: CRITICAL - 1.5% (64684 kB) free! [16:44:12] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [16:44:52] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [16:47:29] @replag all [16:47:30] nosy1: s1-pri: 1s [-0.13 s/s]; s1-sec: 1s [-0.00 s/s]; s1-sec-c: 4s [-0.00 s/s]; s2-pri: 4s [-0.00 s/s]; s2/s5-pri-c: 27m 50s [+0.44 s/s]; s3-rr: 38s [-0.00 s/s]; s3-user: 38s [-0.00 s/s]; s4-rr: 27m 50s [+0.44 s/s] [16:47:31] nosy1: s4-user: error; s5-rr: 1s [-0.00 s/s]; s5-user: 2h 3m 3s [+0.41 s/s]; s6-rr: 3s [-0.01 s/s]; s6-user: 3s [-0.01 s/s]; s7-rr: 1s [-0.01 s/s]; s7-user: 1s [-0.01 s/s] [16:57:53] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1857.000000 [16:58:52] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7578.000000 [16:58:52] Cluster on damiana.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:59:21] SSH on damiana.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:52] ts-array5 on damiana.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:00:52] Environment on damiana.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:00:52] Cluster on damiana.esi is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:00:52] DiskSuite on damiana.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:01:33] SMTP on damiana.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:43] ts-array5 on damiana.esi is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [17:01:53] Cluster on damiana.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:01:53] Cluster on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:06:05] TS down ? [17:09:32] phe: something is wrong [17:10:17] nosy1: ssh to willow is down [17:10:50] nightshade too [17:11:01] and Im seeing IRC bots drop like flys [17:11:14] yeah, the world is coming to an end [17:11:18] this is it guys.. [17:11:28] Nightshade died in the middle of a session and refuses my key. D: [17:11:29] :p [17:11:40] yeah, willow refused my key too [17:11:41] even status.toolserver.org is down [17:11:51] Yup. [17:12:27] we have a problem. [17:12:50] and there goes another [17:13:07] Another one bites the dust. Dun dun dun dun. Another one bites the dust. [17:13:31] * johang is waiting for tsnag to quit [17:14:06] @replag all [17:14:24] nosy1: that bot is down [17:14:50] Ooh, at least with free time to think about it, I just figured out the problem I was working on. Sometimes a break helps. :p [17:15:33] oh yes...damiana is not avail [17:17:22] nosy1: actually, pretty much everything seems to be down [17:17:25] can't log in via ssh [17:17:53] nosy1: i started a job that wants *lots* of ram on willow. let me know if that was causing the problem [17:18:02] can't log in to investigate or kill [17:19:37] Daniel_WMDE_: the graphcore? [17:20:01] yes. [17:20:33] nosy1: kill graphcore/graphserv if they cause issues. [17:20:34] i wondered why you didn't use sge. then it could run an a server with much more ram [17:21:18] Merlissimo: because it's a service. well, a service's child process, the service is graphserv [17:21:28] i need to knowq which ip to connect to :) [17:21:34] but yes, i should pick a different box for that [17:21:50] and not use enwiki for testing :) [17:21:52] sorry. [17:22:10] anyway, time for dinner [17:22:54] hmnm [17:23:02] dinner, that stuff one eats [17:23:40] Daniel_WMDE_: but willow is not the problem, but dns is [17:24:43] hrm [17:24:59] SMF on damiana.esi is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:25:04] SMF on turnera.esi is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:25:33] Free Memory on damiana.esi is OK: OK - 52.0% (2177812 kB) free. [17:25:33] Free Memory on turnera.esi is OK: OK - 71.2% (2979024 kB) free. [17:25:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:25:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42716 MB (4% inode=99%): [17:26:44] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3085.000000 [17:26:54] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.335449/1.75, alarm hl:np_load_avg=0.106445/2.00, alarm hl:mem_free=212.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.335449/1.50, alarm hl:np_load_long=0.125488/1.75, alarm hl:mem_free=212.000000M/250M [17:26:58] everything up again... [17:27:03] Aaron Halfaker * [Toolserver-l] FYI: login.toolserver not responding or asking for password, HTTP down [17:27:04] Cluster on damiana.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, nfs-home Online, check sge-qmaster, [17:27:13] Cluster on turnera.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, nfs-home Online, [17:27:43] NTP on damiana.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.005769 secs [17:27:44] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.007875 secs [17:28:14] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [17:28:14] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [17:29:03] Sumana Harihareswara * Re: [Toolserver-l] FYI: login.toolserver not responding or asking for password, HTTP down [17:30:24] Hm, toolserver.org is up but status.toolserver.org still seems to be down. [17:34:00] AMadman: seems i cannot do anything against this [17:34:02] host status.toolserver.org [17:34:02] status.toolserver.org is an alias for ts-status.tcx.org.uk. [17:34:02] ts-status.tcx.org.uk is an alias for WWW.RT.UK.EU.org. [17:34:02] WWW.RT.UK.EU.org is an alias for TAMARA.RT.UK.EU.org. [17:34:02] TAMARA.RT.UK.EU.org has address 77.75.105.169 [17:34:03] TAMARA.RT.UK.EU.org has IPv6 address 2a01:348:0:6:4d4b:69a9:0:1 [17:34:03] TAMARA.RT.UK.EU.org mail is handled by 20 B.MAIL.RT.UK.EU.org. [17:34:04] TAMARA.RT.UK.EU.org mail is handled by 10 A.MAIL.RT.UK.EU.org. [17:34:04] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [17:34:13] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [17:34:15] Ah, makes sense. [17:34:21] will have to ask dab when he is online again [17:34:29] It'd be pretty silly if it were a part of the Toolserver. :p [17:34:54] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [17:35:44] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [17:38:52] @replag all [17:38:52] nosy1: s1-pri: 27m 47s [-]; s1-sec: 27m 47s [-]; s1-sec-c: 27m 52s [-]; s2-pri: 27m 55s [-]; s2/s5-pri-c: 1h 3m 36s [-]; s3-rr: 28m 11s [-]; s3-user: 28m 11s [-]; s4-rr: 1h 3m 36s [-] [17:38:53] nosy1: s4-user: error; s5-rr: 27m 50s [-]; s5-user: 2h 36m 5s [-]; s6-rr: 27m 49s [-]; s6-user: 27m 49s [-]; s7-rr: 27m 47s [-]; s7-user: 27m 47s [-] [17:40:44] NTP on damiana.esi is OK: NTP OK: Offset 0.000969 secs [17:40:44] NTP on turnera.esi is OK: NTP OK: Offset -0.012152 secs [17:41:21] 3(created) [ACCAPP-452] Research User behavior and user interaction in Wikipedia domain; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-452> (Anuradha Uduwage) [17:42:54] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.350586/1.75, alarm hl:np_load_avg=0.373535/2.00, alarm hl:mem_free=270.000000M/300M [17:43:13] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1927.000000 [17:43:44] s2 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1956.000000 [17:43:54] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1965.000000 [17:43:54] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1967.000000 [17:44:04] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1973.000000 [17:47:34] /sql on thyme is WARNING: DISK WARNING - free space: /sql 195362 MB (20% inode=99%): [17:49:34] Free Memory on damiana.esi is WARNING: WARNING - 19.1% (801172 kB) free! [17:50:23] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default svc:/network/trainwreck:s5 [17:52:34] /sql on thyme is OK: DISK OK - free space: /sql 228496 MB (23% inode=99%): [17:53:24] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2066 [17:54:14] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2433 [17:55:03] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1718.000000 [17:55:03] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1901 [17:55:23] MySQL slave on rosemary is OK: Uptime: 8034574 Threads: 17 Questions: 2409401291 Slow queries: 1008679 Opens: 12019 Flush tables: 1 Open tables: 1015 Queries per second avg: 299.879 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1681 [17:55:54] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1533.000000 [17:56:04] MySQL slave on cassia is OK: Uptime: 5212375 Threads: 7 Questions: 843273135 Slow queries: 33317 Opens: 607 Flush tables: 1 Open tables: 269 Queries per second avg: 161.782 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1300 [17:56:53] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2576 [17:57:13] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2668 [17:57:34] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2673 [17:58:24] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2751 [18:00:14] MySQL slave on thyme is OK: Uptime: 2986186 Threads: 28 Questions: 1035559134 Slow queries: 514221 Opens: 148008 Flush tables: 1 Open tables: 2803 Queries per second avg: 346.783 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1787 [18:01:04] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1720.000000 [18:05:53] MySQL slave on daphne is OK: Uptime: 6462504 Threads: 40 Questions: 5364015004 Slow queries: 834645 Opens: 35148425 Flush tables: 3 Open tables: 16338 Queries per second avg: 830.21 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1791 [18:06:44] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1678.000000 [18:11:24] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [18:12:44] SMF on rosemary is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [18:23:14] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.091797/1.00, alarm hl:np_load_long=0.676758/1.50, alarm hl:mem_free=20852.000000M/300M [18:24:14] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [18:25:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42511 MB (4% inode=99%): [18:25:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:25:45] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [18:25:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:25:54] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [18:25:54] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:25:54] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12182.000000 [18:32:03] Sumana Harihareswara * [Toolserver-l] Save the date: Wikimedia & MediaWiki hackathon in Berlin, 1-3 June 2012 [18:35:45] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3626 [18:35:46] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7226.000000 [18:42:25] MySQL slave on z-dat-s7-a is OK: Uptime: 115789 Threads: 14 Questions: 11563888 Slow queries: 4679 Opens: 60851 Flush tables: 1 Open tables: 3079 Queries per second avg: 99.870 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1762 [18:47:24] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3618 [18:49:34] Free Memory on damiana.esi is WARNING: WARNING - 14.1% (590704 kB) free! [18:50:34] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default svc:/network/trainwreck:s5 [18:57:18] [[Special:Log/newusers]] create 10 * Glove * (New user account) [18:59:24] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3571 [19:09:24] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [19:11:24] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7218.000000 [19:12:44] SMF on rosemary is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [19:14:44] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3516 [19:21:44] MySQL slave on z-dat-s6-a is OK: Uptime: 178521 Threads: 3 Questions: 14514515 Slow queries: 10782 Opens: 99716 Flush tables: 1 Open tables: 1779 Queries per second avg: 81.304 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1709 [19:25:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43283 MB (4% inode=99%): [19:25:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:25:44] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [19:25:54] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [19:25:54] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:25:54] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15781.000000 [19:26:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:36:43] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10886.000000 [19:40:34] Free Memory on damiana.esi is CRITICAL: CRITICAL - 9.4% (392052 kB) free! [19:46:34] Free Memory on damiana.esi is WARNING: WARNING - 10.2% (425304 kB) free! [19:50:33] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default svc:/network/trainwreck:s5 [19:54:34] Free Memory on damiana.esi is CRITICAL: CRITICAL - 9.9% (414720 kB) free! [20:09:24] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [20:11:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10819.000000 [20:13:03] SMF on rosemary is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [20:19:34] Free Memory on damiana.esi is CRITICAL: CRITICAL - 7.8% (328240 kB) free! [20:21:34] SMF on daphne is OK: OK - all services online [20:22:24] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 17925 MB (1% inode=99%): [20:24:04] SMF on rosemary is OK: OK - all services online [20:25:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 43092 MB (4% inode=99%): [20:25:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:25:54] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [20:26:04] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [20:26:04] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19155.000000 [20:26:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:27:03] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:33:24] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [20:34:18] nosy1: tcx.org.uk is river. [20:34:23] we should not be runnign anything there [20:34:38] we should find a new solution for status.toolserver.org [20:34:55] quick stop gap would be a protected page on the toolserver wiki [20:35:03] just change the dns, set up a http redirect... [20:37:04] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13740.000000 [20:44:23] MySQL on z-dat-s3-a is OK: Uptime: 550 Threads: 9 Questions: 17932 Slow queries: 3 Opens: 304 Flush tables: 1 Open tables: 278 Queries per second avg: 32.603 [20:53:24] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3415.000000 [20:58:24] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1480.000000 [21:09:34] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [21:19:43] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.7% (237764 kB) free! [21:25:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42853 MB (4% inode=99%): [21:26:03] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [21:26:03] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [21:26:14] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:26:14] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20940.000000 [21:26:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:27:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:30:34] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3560 [21:37:14] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14910.000000 [21:41:34] MySQL slave on z-dat-s3-a is OK: Uptime: 1455 Threads: 23 Questions: 648600 Slow queries: 46 Opens: 8435 Flush tables: 1 Open tables: 8424 Queries per second avg: 445.773 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1575 [22:19:44] Free Memory on damiana.esi is CRITICAL: CRITICAL - 3.5% (144808 kB) free! [22:25:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42585 MB (4% inode=99%): [22:26:14] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [22:26:14] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [22:26:14] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20969.000000 [22:26:14] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:26:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:27:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:37:14] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14040.000000 [23:19:44] Free Memory on damiana.esi is CRITICAL: CRITICAL - 2.5% (102836 kB) free! [23:25:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42520 MB (4% inode=99%): [23:26:14] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:26:14] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:26:14] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:26:14] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22883.000000 [23:26:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:27:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:37:14] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14709.000000