[00:03:01] Load avg. on willow is WARNING: WARNING - load average: 15.37, 14.21, 11.59 [00:04:02] Load avg. on willow is OK: OK - load average: 14.43, 14.17, 11.75 [00:41:01] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [00:41:12] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [00:41:12] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [00:41:12] SMTP on yarrow is CRITICAL: Connection refused [00:41:21] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:41:21] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:41:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:41:31] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:41:31] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:41:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:41:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:42:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:42:32] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:42:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:53:23] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [00:53:23] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [00:55:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [00:55:32] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [00:55:52] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1912.000000 [01:03:12] Load avg. on willow is WARNING: WARNING - load average: 15.50, 15.44, 13.40 [01:18:12] Load avg. on willow is OK: OK - load average: 12.33, 14.55, 14.54 [01:31:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [01:41:01] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [01:41:32] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [01:41:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:42:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:42:11] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [01:42:11] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [01:42:11] SMTP on yarrow is CRITICAL: Connection refused [01:42:21] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:42:21] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:42:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:42:31] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:42:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:42:41] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [01:42:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:54:22] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [01:54:22] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [01:55:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [01:55:31] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [02:13:11] Load avg. on willow is WARNING: WARNING - load average: 15.10, 15.27, 14.21 [02:15:11] Load avg. on willow is OK: OK - load average: 13.91, 14.86, 14.20 [02:32:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5810.000000 [02:33:11] Load avg. on willow is WARNING: WARNING - load average: 14.57, 15.57, 14.83 [02:37:11] Load avg. on willow is OK: OK - load average: 13.09, 14.80, 14.71 [02:41:01] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [02:41:31] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:41:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:42:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:42:11] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [02:42:11] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [02:42:11] SMTP on yarrow is CRITICAL: Connection refused [02:42:21] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:42:22] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:42:31] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:42:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:42:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:42:41] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:42:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:43:11] Load avg. on willow is WARNING: WARNING - load average: 14.11, 15.18, 14.92 [02:55:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [02:55:21] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [02:55:22] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [02:56:31] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [03:16:12] Load avg. on willow is WARNING: WARNING - load average: 16.75, 15.38, 14.79 [03:20:11] Load avg. on willow is OK: OK - load average: 14.44, 14.90, 14.73 [03:25:02] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3137.000000 [03:33:10] Load avg. on willow is WARNING: WARNING - load average: 15.65, 15.58, 14.98 [03:37:02] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1478.000000 [03:39:11] Load avg. on willow is OK: OK - load average: 12.88, 14.55, 14.80 [03:41:02] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [03:41:42] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:41:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:42:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:42:11] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [03:42:11] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [03:42:22] SMTP on yarrow is CRITICAL: Connection refused [03:42:43] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:42:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:42:43] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:42:43] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:42:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:43:21] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:43:21] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:55:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [03:55:22] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [03:55:31] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [03:56:43] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [04:13:11] Load avg. on willow is WARNING: WARNING - load average: 15.02, 15.21, 15.37 [04:18:11] Load avg. on willow is OK: OK - load average: 12.15, 14.11, 14.89 [04:41:01] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [04:41:54] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:41:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:42:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:42:11] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [04:42:12] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [04:42:22] SMTP on yarrow is CRITICAL: Connection refused [04:42:54] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:42:54] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:42:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:42:54] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:42:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:43:21] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:43:22] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:55:22] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [04:55:32] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [04:56:02] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [04:56:53] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [05:03:11] Load avg. on willow is WARNING: WARNING - load average: 14.74, 15.19, 14.45 [05:04:12] Load avg. on willow is OK: OK - load average: 13.86, 14.85, 14.38 [05:13:11] Load avg. on willow is WARNING: WARNING - load average: 15.49, 15.46, 14.75 [05:19:07] [[Category:Edit counters]] ! 10https://wiki.toolserver.org/w/index.php?diff=6698&oldid=6684&rcid=8845 * 94.167.184.158 * (+35) (italian added) [05:28:02] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 8358 MB (0% inode=99%): [05:41:02] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [05:41:53] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:41:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:42:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:42:11] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [05:42:22] SMTP on yarrow is CRITICAL: Connection refused [05:42:55] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:42:55] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:42:55] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:42:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:42:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:43:12] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [05:43:21] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:43:21] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:51:22] 3(commented) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (Marlen Caemmerer) [05:55:22] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [05:55:31] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [05:56:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [05:56:54] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [06:03:12] Load avg. on willow is WARNING: WARNING - load average: 17.02, 17.18, 15.34 [06:41:01] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [06:42:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:42:21] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [06:42:52] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:42:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:43:12] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [06:43:21] SMTP on yarrow is CRITICAL: Connection refused [06:43:22] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:43:22] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:43:53] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:43:53] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:43:53] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:43:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:43:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:56:02] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [06:56:21] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [06:56:30] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [06:57:53] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [07:04:12] Load avg. on willow is WARNING: WARNING - load average: 14.29, 16.50, 16.46 [07:41:07] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [07:42:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [07:42:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:43:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:43:39] SMTP on yarrow is CRITICAL: Connection refused [07:43:39] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [07:43:40] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:43:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:43:48] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:44:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:44:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:44:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:44:48] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:44:48] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:56:40] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [07:56:40] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [07:56:40] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [07:58:48] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [08:04:38] Load avg. on willow is WARNING: WARNING - load average: 14.78, 15.38, 15.96 [08:13:39] Load avg. on willow is OK: OK - load average: 13.71, 13.96, 14.91 [08:41:07] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [08:42:40] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [08:42:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:43:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:44:38] SMTP on yarrow is CRITICAL: Connection refused [08:44:38] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [08:44:38] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:44:38] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:44:47] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:44:59] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:45:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:45:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:45:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:45:47] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:56:39] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [08:56:40] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [08:56:40] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [08:57:37] Load avg. on willow is WARNING: WARNING - load average: 18.61, 15.38, 14.25 [08:59:47] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [09:00:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 40.54, 21.95, 16.78 [09:02:40] Load avg. on willow is WARNING: WARNING - load average: 29.30, 24.57, 18.46 [09:09:31] hi [09:28:39] Load avg. on willow is OK: OK - load average: 13.07, 13.79, 14.98 [09:37:32] is there any list of ip addresses of toolserver machines? [09:41:08] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [09:43:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:43:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:43:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [09:44:40] SMTP on yarrow is CRITICAL: Connection refused [09:44:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:44:40] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:44:40] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [09:44:47] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:45:00] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:45:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:45:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:45:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:45:47] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [09:57:39] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [09:57:39] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [09:57:39] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [10:00:47] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [10:02:38] Load avg. on willow is WARNING: WARNING - load average: 18.54, 16.99, 15.30 [10:08:39] Load avg. on willow is OK: OK - load average: 13.09, 14.89, 14.92 [10:12:38] Load avg. on willow is WARNING: WARNING - load average: 16.35, 15.65, 15.19 [10:41:08] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [10:42:39] Load avg. on willow is WARNING: WARNING - load average: 15.04, 15.43, 14.77 [10:43:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:43:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [10:43:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:44:47] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:45:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:45:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:45:38] SMTP on yarrow is CRITICAL: Connection refused [10:45:38] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:45:38] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:45:38] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [10:45:58] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:45:58] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:46:47] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:47:40] Load avg. on willow is OK: OK - load average: 13.02, 14.73, 14.71 [10:57:38] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [10:57:39] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [10:57:39] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [11:00:47] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [11:02:39] Load avg. on willow is WARNING: WARNING - load average: 21.40, 19.28, 16.45 [11:25:23] [[Special:Log/newusers]] create 10 * Euriditi * (New user account) [11:37:37] Load avg. on willow is OK: OK - load average: 12.70, 13.71, 14.88 [11:41:18] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [11:43:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:43:59] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [11:44:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:44:48] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [11:45:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:45:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:45:38] SMTP on yarrow is CRITICAL: Connection refused [11:45:38] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:45:38] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:45:48] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [11:45:58] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [11:46:07] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:46:47] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:52:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41112 MB (4% inode=99%): [11:57:47] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [11:58:38] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [11:58:38] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [12:00:47] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [12:02:37] Load avg. on willow is WARNING: WARNING - load average: 20.35, 19.12, 16.14 [12:22:38] Load avg. on willow is OK: OK - load average: 12.14, 13.90, 14.87 [12:32:47] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196345 MB (20% inode=99%): [12:41:17] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [12:43:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:43:59] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [12:44:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:45:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:45:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:45:38] SMTP on yarrow is CRITICAL: Connection refused [12:45:38] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:45:38] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:45:48] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [12:45:48] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [12:45:58] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [12:46:18] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:46:48] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:57:49] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [12:59:38] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [12:59:38] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [13:00:48] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [13:34:49] /sql on thyme is OK: DISK OK - free space: /sql 221464 MB (23% inode=99%): [13:43:18] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [13:43:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:43:59] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [13:44:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:45:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:45:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:45:39] SMTP on yarrow is CRITICAL: Connection refused [13:45:39] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:45:39] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:45:48] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [13:45:48] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [13:45:58] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [13:46:27] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:46:48] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:58:47] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [13:59:39] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [13:59:39] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [14:00:48] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [14:41:38] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [14:43:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:44:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:44:58] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [14:45:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:45:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:45:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:45:49] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:45:49] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [14:45:49] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [14:46:28] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:46:38] SMTP on yarrow is CRITICAL: Connection refused [14:46:49] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:46:57] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [14:49:16] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1956.000000 [14:57:08] s4 replag on cassia is CRITICAL: (Service Check Timed Out) [14:57:17] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2427.000000 [14:58:48] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [14:59:47] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [15:00:37] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [15:00:48] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [15:14:46] nosy: Are you around? [15:42:37] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [15:43:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:44:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:44:57] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [15:45:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:45:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:46:48] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:46:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:46:48] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [15:46:48] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:46:48] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:46:57] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:47:27] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:47:37] SMTP on yarrow is CRITICAL: Connection refused [15:57:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2836.000000 [15:58:48] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [15:59:49] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [16:00:36] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [16:00:47] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [16:42:37] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [16:43:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:44:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:44:59] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [16:45:18] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1685.000000 [16:46:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:46:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:46:59] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:46:59] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [16:46:59] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [16:46:59] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [16:47:27] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:47:37] SMTP on yarrow is CRITICAL: Connection refused [16:47:48] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:47:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:56:27] 3(commented) [OSM-3] Ptolemy Postgres crashed twice <10https://jira.toolserver.org/browse/OSM-3> (Kai Krueger) [16:58:47] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [16:59:49] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [17:00:36] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [17:00:48] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [17:05:06] i [17:28:47] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 8357 MB (0% inode=99%): [17:42:37] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [17:44:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:44:37] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:45:08] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [17:46:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:47:08] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [17:47:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:47:37] SMTP on yarrow is CRITICAL: Connection refused [17:47:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:47:48] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:47:58] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:47:58] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [17:47:58] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [17:48:26] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:59:48] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [18:00:37] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [18:00:47] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [18:00:58] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [18:12:57] Load avg. on willow is WARNING: WARNING - load average: 21.50, 16.47, 13.21 [18:19:57] Load avg. on willow is OK: OK - load average: 9.99, 14.61, 13.89 [18:42:42] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [18:44:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:45:34] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [18:45:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:46:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:47:33] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [18:47:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:47:42] SMTP on yarrow is CRITICAL: Connection refused [18:47:52] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:47:52] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:48:02] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [18:48:34] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:48:53] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [18:48:53] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [18:59:53] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [19:00:42] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [19:00:52] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [19:01:52] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [19:43:00] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [19:44:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:45:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:46:29] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [19:46:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:47:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:48:00] SMTP on yarrow is CRITICAL: Connection refused [19:48:00] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:48:00] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:48:29] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [19:48:40] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:49:00] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [19:49:00] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:49:01] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [20:00:00] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [20:01:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [20:01:01] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [20:02:01] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [20:41:20] 3(created) [UTRS-56] Duplicate action allowed, should disable 2nd action instead; UTRS; Minor Bug <10https://jira.toolserver.org/browse/UTRS-56> (DeltaQuad) [20:43:10] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [20:44:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:46:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:46:29] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [20:46:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:47:59] SMTP on yarrow is CRITICAL: Connection refused [20:47:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:47:59] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:48:00] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:48:29] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [20:49:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:49:10] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [20:49:10] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:49:10] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [20:59:59] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [21:00:37] nrape? [21:02:00] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [21:02:00] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [21:02:00] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [21:08:29] NRPE??? [21:09:18] * valhallasw hits matthewrbowker with google [21:09:25] it is super effective [21:10:00] (NRPE is, apparently, a plugin that allows you to run tests on remote servers) [21:10:05] valhallasw: My browser was lagging.......... [21:10:53] ah. valhallasw: thanks :) [21:19:48] [[Special:Log/newusers]] create 10 * KhabarNegar * (New user account) [21:43:10] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [21:44:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:45:52] any TS admin here? [21:45:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:46:30] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [21:46:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:47:59] SMTP on yarrow is CRITICAL: Connection refused [21:47:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:47:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:47:59] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:48:30] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [21:49:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:49:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [21:49:09] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:49:09] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [21:59:59] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [22:02:00] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [22:03:00] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [22:03:00] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [22:43:10] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [22:45:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:46:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:46:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [22:46:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:48:00] SMTP on yarrow is CRITICAL: Connection refused [22:48:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:48:10] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:48:10] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:48:29] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [22:48:59] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:49:10] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:49:10] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [22:49:10] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [23:00:10] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [23:02:01] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [23:03:10] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [23:03:11] Environment on yarrow is UNKNOWN: NRPE: Unable to read output [23:28:10] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.007812/1.00, alarm hl:np_load_long=0.548828/1.50, alarm hl:mem_free=21764.000000M/300M, alarm hl:available=1/0 [23:29:10] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [23:43:19] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [23:45:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:46:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:46:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [23:47:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:47:59] SMTP on yarrow is CRITICAL: Connection refused [23:47:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:48:10] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:48:10] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:48:29] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [23:49:09] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:49:10] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [23:49:10] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [23:50:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:53:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40325 MB (4% inode=99%):