[00:02:14] Load avg. on willow is WARNING: WARNING - load average: 18.00, 15.97, 12.36 [00:03:23] Load avg. on willow is OK: OK - load average: 11.56, 14.50, 12.08 [00:09:14] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [00:09:55] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [00:11:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:12:32] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [00:12:55] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:12:56] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [00:16:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:17:14] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [00:22:55] /aux0 on yarrow is CRITICAL: Connection refused by host [00:22:55] Load avg. on yarrow is CRITICAL: Connection refused by host [00:22:55] Environment on yarrow is CRITICAL: Connection refused by host [00:22:55] / on yarrow is CRITICAL: Connection refused by host [00:23:02] NTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:23] SMTP on yarrow is CRITICAL: Connection refused [00:23:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:23:44] s4 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [00:23:44] s2 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [00:24:43] MySQL slave on yarrow is CRITICAL: Cant connect to MySQL server on yarrow (146) [00:24:43] MySQL on yarrow is CRITICAL: Cant connect to MySQL server on yarrow (146) [00:24:43] s5 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [00:25:14] Sun Grid Engine execd on yarrow is CRITICAL: Connection refused by host [00:25:14] /tmp on yarrow is CRITICAL: Connection refused by host [00:25:14] SMF on yarrow is CRITICAL: Connection refused by host [00:25:14] Cluster on yarrow is CRITICAL: Connection refused by host [00:52:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:09:14] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [01:10:03] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [01:11:23] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [01:11:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:12:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [01:13:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:13:03] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [01:16:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:17:14] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [01:42:02] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1971.000000 [01:46:15] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1912.000000 [01:47:14] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1913.000000 [01:52:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:02:24] Load avg. on willow is WARNING: WARNING - load average: 16.36, 15.10, 12.45 [02:03:23] Load avg. on willow is OK: OK - load average: 12.32, 14.21, 12.31 [02:09:15] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [02:10:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3657.000000 [02:10:15] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [02:12:23] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [02:12:23] Load avg. on willow is WARNING: WARNING - load average: 18.14, 15.57, 13.57 [02:12:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:12:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [02:13:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:13:14] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [02:16:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:17:15] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [02:32:14] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1169.000000 [02:34:15] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1286.000000 [02:52:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:02:23] Load avg. on willow is WARNING: WARNING - load average: 16.59, 15.90, 14.22 [03:07:34] Load avg. on willow is OK: OK - load average: 13.60, 14.88, 14.23 [03:10:14] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [03:11:15] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7317.000000 [03:11:15] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [03:12:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:12:33] Load avg. on willow is WARNING: WARNING - load average: 15.83, 15.77, 14.74 [03:12:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:12:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [03:13:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:13:24] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [03:17:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:18:14] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [03:24:36] [[Special:Log/newusers]] create 10 * Smy987 * (New user account) [03:49:21] 3(created) [TS-1308] Install the XML_Serializer PEAR package (PHP); Toolserver: Software installation; Task <10https://jira.toolserver.org/browse/TS-1308> (Jesse Plamondon-Willard) [03:52:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:10:14] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1899.000000 [04:10:14] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [04:11:14] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1959.000000 [04:12:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10977.000000 [04:12:15] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [04:12:34] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:12:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:12:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [04:13:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:13:23] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [04:17:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:18:23] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [04:30:12] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6688&oldid=6667&rcid=8834 * 91.198.174.202 * (+4) (updated page) [04:46:14] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [04:47:14] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [04:53:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:10:23] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [05:12:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [05:13:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14638.000000 [05:13:13] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [05:13:13] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:13:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:13:44] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [05:13:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:17:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:19:23] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [05:24:23] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 7426 MB (0% inode=99%): [05:46:14] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6312.000000 [05:48:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6361.000000 [05:53:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:10:23] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [06:12:43] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [06:13:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18238.000000 [06:13:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:13:14] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [06:13:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:13:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:13:43] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [06:17:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:19:23] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [06:46:14] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9290.000000 [06:48:14] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9338.000000 [06:53:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:10:23] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [07:12:33] Load avg. on willow is WARNING: WARNING - load average: 17.34, 15.08, 12.95 [07:12:44] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [07:13:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21837.000000 [07:13:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:13:14] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [07:13:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [07:13:53] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [07:14:14] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3255.000000 [07:14:33] Load avg. on willow is OK: OK - load average: 13.93, 14.58, 13.03 [07:14:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:17:15] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 958.000000 [07:18:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:19:00] @replag all [07:19:01] nosy: s1-pri: 2s [+0.00 s/s]; s1-sec: 2s [+0.00 s/s]; s1-sec-c: 1s [-]; s2-pri: 1m 8s [+0.00 s/s]; s2/s5-pri-c: 2h 8m 41s [+0.09 s/s]; s3-rr: 54s [-0.00 s/s]; s3-user: 54s [-0.00 s/s]; s4-rr: 2h 8m 41s [+0.09 s/s] [07:19:02] nosy: s4-user: 6h 9m 56s [+0.35 s/s]; s5-rr: 5s [+0.00 s/s]; s5-user: 5s [+0.00 s/s]; s6-rr: 5s [+0.00 s/s]; s6-user: 5s [+0.00 s/s]; s7-rr: 1s [-0.00 s/s]; s7-user: 1s [-0.00 s/s] [07:20:24] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [07:26:08] is nightshade down? [07:32:10] yes [07:32:30] seems to have lost its hard disks somehow [07:32:38] @replag all [07:32:39] nosy: s1-pri: 2s [-]; s1-sec: 2s [-]; s1-sec-c: 5m 29s [+0.40 s/s]; s2-pri: 21s [-0.06 s/s]; s2/s5-pri-c: 1h 31m 5s [-2.76 s/s]; s3-rr: 1m 29s [+0.04 s/s]; s3-user: 1m 29s [+0.04 s/s]; s4-rr: 5m 29s [-9.04 s/s] [07:32:40] nosy: s4-user: 6h 23m 33s [+1.00 s/s]; s5-rr: 1s [-0.00 s/s]; s5-user: 1s [-0.00 s/s]; s6-rr: 2s [-0.00 s/s]; s6-user: 2s [-0.00 s/s]; s7-rr: 8s [+0.01 s/s]; s7-user: 8s [+0.01 s/s] [07:33:20] too bad [07:33:25] what about the data? [07:33:57] the user data is on an array which is to find on ig willow too [07:34:13] but the cron jobs themselve might be lost [07:34:31] sigh [07:34:44] glad I have a backup [07:46:15] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [07:53:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:59:13] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1922.000000 [08:02:33] Load avg. on willow is WARNING: WARNING - load average: 15.21, 14.04, 12.56 [08:03:34] Load avg. on willow is OK: OK - load average: 13.86, 13.86, 12.59 [08:10:24] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [08:12:53] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [08:13:24] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:13:24] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [08:13:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:13:53] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [08:14:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25502.000000 [08:14:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:20:24] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [08:24:22] 3(commented) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (Marlen Caemmerer) [08:27:25] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [08:47:13] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7269.000000 [08:53:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:10:33] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [09:12:53] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [09:13:23] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:13:23] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [09:13:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:14:13] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29103.000000 [09:14:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:14:53] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [09:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:20:33] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [09:28:24] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7268.000000 [09:40:07] @replag [09:40:08] Quentinv57: s1-sec-c: 2h 12m 59s [+1.00 s/s]; s2/s5-pri-c: 2h 54m 7s [+0.65 s/s]; s3-rr: 4m 58s [+0.03 s/s]; s3-user: 4m 58s [+0.03 s/s]; s4-rr: 2h 12m 59s [+1.00 s/s]; s4-user: 8h 31m 3s [+1.00 s/s]; s6-rr: 48s [+0.01 s/s]; s6-user: 48s [+0.01 s/s] [09:40:09] Quentinv57: s7-rr: 53s [+0.01 s/s]; s7-user: 53s [+0.01 s/s] [09:47:19] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10870.000000 [09:52:39] Load avg. on willow is WARNING: WARNING - load average: 15.38, 14.39, 12.10 [09:53:38] Load avg. on willow is OK: OK - load average: 12.57, 13.79, 12.04 [09:53:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:00:28] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:00:29] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:01:09] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:01:19] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:08:30] MySQL on daphne is OK: Uptime: 599 Threads: 35 Questions: 137 Slow queries: 2 Opens: 27 Flush tables: 1 Open tables: 18 Queries per second avg: 0.228 [10:08:57] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 593.000000 [10:09:19] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 663.000000 [10:09:29] MySQL slave on daphne is OK: Uptime: 659 Threads: 36 Questions: 769 Slow queries: 12 Opens: 223 Flush tables: 1 Open tables: 204 Queries per second avg: 1.166 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 659 [10:11:29] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [10:11:38] Load avg. on willow is WARNING: WARNING - load average: 24.39, 18.13, 14.54 [10:13:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [10:13:28] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:13:28] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [10:14:18] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32704.000000 [10:14:38] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:15:09] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [10:15:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:19:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:20:39] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [10:28:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10870.000000 [10:38:29] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3500.000000 [10:48:29] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1619.000000 [10:53:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:11:29] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [11:13:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [11:13:28] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:13:29] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [11:14:17] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36307.000000 [11:14:38] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:16:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:16:08] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [11:19:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:21:38] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [11:28:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14475.000000 [11:50:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 44727 MB (4% inode=99%): [11:53:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:02:38] Load avg. on willow is WARNING: WARNING - load average: 19.95, 16.77, 13.19 [12:04:40] Load avg. on willow is OK: OK - load average: 10.57, 14.33, 12.72 [12:08:39] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3555.000000 [12:11:39] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [12:12:40] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1351.000000 [12:13:40] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:13:40] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [12:14:08] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [12:14:18] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39907.000000 [12:14:39] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:16:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:16:19] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [12:20:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:21:40] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [12:53:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:11:39] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [13:13:40] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [13:13:40] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:14:39] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:15:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [13:15:18] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43569.000000 [13:16:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:16:18] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [13:20:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:21:49] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [13:33:12] can I receive mail when jobs start/finish using sh instead of python? [13:53:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:05:18] nosy: ping [14:11:49] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [14:13:49] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [14:13:49] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:14:16] [[Special:Log/newusers]] create 10 * YOUSSOUFJUSUF * (New user account) [14:14:50] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:15:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47174.000000 [14:16:08] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [14:16:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:16:18] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [14:20:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:21:50] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [14:54:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:11:59] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [15:13:59] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [15:13:59] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:15:00] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [15:15:28] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50775.000000 [15:16:17] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [15:16:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:16:18] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [15:20:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:21:59] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [15:48:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:48:48] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:48:59] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:48:59] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:48:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:54:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:58:18] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/application/ts/mysql-51:default [16:00:17] SMF on daphne is OK: OK - all services online [16:12:59] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [16:14:00] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [16:14:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:15:00] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [16:15:29] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54378.000000 [16:16:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:16:28] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [16:16:28] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [16:20:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:22:00] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [16:32:39] what happened to the toolserver? :( [16:34:04] http://en.wikipedia.org/wiki/Wikipedia:Huggle/Members [16:34:08] sorry [16:34:12] wrong channel [16:42:28] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/application/ts/mysql-51:default [16:48:28] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:48:49] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [16:48:59] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [16:48:59] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:49:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [16:54:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:13:20] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [17:14:20] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [17:14:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:14:59] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [17:15:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57988.000000 [17:16:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:16:40] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [17:16:40] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [17:20:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:22:19] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [17:25:00] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 103 MB (0% inode=99%): [17:42:29] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/application/ts/mysql-51:default [17:48:29] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:48:50] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [17:49:21] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [17:49:21] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:50:20] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [17:54:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:57:29] SMF on daphne is OK: OK - all services online [18:13:20] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [18:14:20] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [18:14:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:15:21] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [18:16:38] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61650.000000 [18:17:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:17:40] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [18:17:40] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [18:17:49] MySQL on daphne is OK: Uptime: 10 Threads: 4 Questions: 282 Slow queries: 0 Opens: 58 Flush tables: 1 Open tables: 49 Queries per second avg: 28.200 [18:20:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:22:21] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [18:23:50] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [18:25:50] MySQL on daphne is OK: Uptime: 14 Threads: 4 Questions: 53 Slow queries: 0 Opens: 39 Flush tables: 1 Open tables: 32 Queries per second avg: 3.785 [18:48:31] s2 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14044.000000 [18:50:19] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [18:50:19] s5 replag on daphne is CRITICAL: (Return code of 139 is out of bounds) [18:51:19] s4 replag on daphne is CRITICAL: (Return code of 139 is out of bounds) [18:54:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:07:03] Marlen Caemmerer * [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [19:07:21] 3(updated) [MNT-1202] Add more space to daphne <10https://jira.toolserver.org/browse/MNT-1202> (Marlen Caemmerer) [19:07:22] 3(created) [MNT-1202] Add more space to daphne; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1202> (Marlen Caemmerer) [19:12:49] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [19:13:30] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [19:14:30] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [19:14:41] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:16:19] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [19:16:42] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65252.000000 [19:17:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:17:41] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [19:17:50] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [19:20:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:22:41] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [19:30:03] Michael Movchin * Re: [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [19:45:49] FC 0/7 on fsw1-n1-oe16-esams.mgmt is OK: FC port 0/7:DOWN:1 UP: OK [19:48:02] Marlen Caemmerer * Re: [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [19:50:41] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [19:50:41] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:50:49] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [19:50:59] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:51:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [19:52:10] I had my crontab on login.toolserver.org [19:52:15] Where can I find it now? [19:53:03] Michael Movchin * Re: [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [19:54:30] vvv: maybe you have to wait: see https://jira.toolserver.org/browse/MNT-1198 to problem with one of the login server. [19:54:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:55:59] Quedel: looks like that [19:59:01] i need some help. I have a clone of Watchlist-bots, but if i put them running on longrun with cronie, i get the error: "NameError: global name 'bot1' is not defined" in bot1.die() [19:59:12] but on cmd, it works fine [20:00:07] the original source code is here: https://svn.toolserver.org/svnroot/p_stewardbots/trunk/Watchlist-bots/BLWatcher.py [20:00:23] different library paths? might not be that your cron-job has the same paths defined [20:00:51] hmmm... [20:00:52] wait [20:01:08] that would probably lead to an import error instead, so disregard my suggestion [20:01:42] in my cron tab i use: "0 * * * * cronsub -sl albeth python $HOME/ircBots/bibliobot.py" [20:03:06] which line is throwing the NameError? I see a call to 'bot1.die()' in the exception handler, but bot1 might not at that point be defined [20:04:23] (bottom of the file) [20:04:33] that's the case, on the exception handeler [20:04:55] Also keep in mind it'll be running in a temp directory, I found. [20:05:06] So you might need to change the working directory before you import files and such. [20:05:50] so i need to use a bash script to change the working directory? [20:07:05] Alchimista: I find a sys.path.append can avoid those issues [20:07:52] Betacommand: ok, i'll try it [20:13:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [20:13:40] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [20:14:41] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [20:14:41] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:16:19] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [20:16:49] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68845.000000 [20:17:50] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [20:18:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:20:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:20:55] Betacommand: a sys.path.append to the directory where the script is? That doesn't work, i got the same error [20:22:51] Alchimista: at which path do you start it from command line? [20:23:32] where the bot scripts are [20:23:41] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [20:24:15] so $home/ircBots ? [20:24:19] Merlissimo: i start it on "/home/alchimista/ircBots", and added to the py file: import sys sys.path.append('/home/alchimista/ircBots') [20:24:59] does it needs the $ on solaris? [20:26:14] Alchimista: does it run if you use: "qsub -wd /home/alchimista/ircBots -N albeth -b /opt/ts/python/2.7/bin/python /home/alchimista/ircBots/bibliobot.py" [20:27:10] Merlissimo: qsub: invalid option argument "-b /opt/ts/python/2.7/bin/python" [20:27:40] ups -b y : "qsub -wd /home/alchimista/ircBots -N albeth -b y /opt/ts/python/2.7/bin/python /home/alchimista/ircBots/bibliobot.py" [20:28:11] yap, worked [20:29:31] Merlissimo: so what's worng with the script/cronie? [20:31:24] two possibilities: your scripts does not like to be started from you homedir, that why added the -wd option. second i think python executable whould not be copied (that's the -b y) [20:32:00] how long does you script runs normally? [20:32:33] it's an irc script, so it's supose to run 24h per day [20:36:10] then you should also add "-h_rt=INFINITY -r y " [20:36:42] on qsub? That way i won't need cronie? [20:37:09] MySQL on daphne is OK: Uptime: 50 Threads: 8 Questions: 346 Slow queries: 1 Opens: 84 Flush tables: 1 Open tables: 77 Queries per second avg: 6.920 [20:37:17] yes [20:37:50] there will be an replacement for cronsub soon [20:38:16] because cronsub does not allow all arguments. [20:39:55] for some scripts i use some bash, this is the only one i couldn't put on cronie [20:42:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [20:46:29] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.652 second response time [20:49:19] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 10.786 second response time [20:50:50] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [20:51:09] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:51:40] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [20:51:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:51:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [20:54:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:12:21] 3(commented) [DRTRIGON-114] Support for named groups in regexs <10https://jira.toolserver.org/browse/DRTRIGON-114> (drtrigon) [21:14:40] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [21:14:50] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:15:41] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [21:16:19] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42666 [21:16:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42099.000000 [21:17:59] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [21:18:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:20:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:23:40] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [21:29:50] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [21:42:09] MySQL on daphne is OK: Uptime: 13 Threads: 2 Questions: 13 Slow queries: 0 Opens: 18 Flush tables: 1 Open tables: 11 Queries per second avg: 1.0 [21:47:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [21:49:09] MySQL on daphne is OK: Uptime: 34 Threads: 16 Questions: 238 Slow queries: 0 Opens: 108 Flush tables: 1 Open tables: 101 Queries per second avg: 7.0 [21:50:51] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [21:51:09] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:52:39] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:52:39] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [21:52:39] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [21:54:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:11:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [22:14:52] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [22:14:52] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:15:50] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [22:16:19] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7552 [22:16:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7590.000000 [22:17:58] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [22:18:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:20:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:52] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [22:27:59] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3359.000000 [22:28:19] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3074 [22:29:52] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [22:30:19] MySQL slave on z-dat-s4-a is OK: Uptime: 1237565 Threads: 10 Questions: 44359310 Slow queries: 56310 Opens: 9528 Flush tables: 1 Open tables: 424 Queries per second avg: 35.844 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1658 [22:30:52] Hey I'm getting some weird outputs at https://toolserver.org/~luxo/contributions/contributions.php?user=Sven+Manguard&blocks=true and https://toolserver.org/~quentinv57/sulinfo/Sven_Manguard [22:30:59] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1227.000000 [22:30:59] it's saying servers are down and stuff [22:32:29] databases are down? Luxo's global user contribution tool says it cannot connect all databases. [22:36:32] pathoschild's stalktoy is also having an SQL connection problem [22:50:59] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [22:51:09] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:52:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:52:40] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [22:52:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [22:55:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:58:05] AccountEligibilitybeta is also down [23:01:37] mys_721tx: yes - s2 and s5-user failed completely [23:02:02] Marlen Caemmerer * Re: [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [23:07:20] 3(commented) [MNT-1202] Add more space to daphne <10https://jira.toolserver.org/browse/MNT-1202> (Marlen Caemmerer) [23:10:25] 3(created) [TS-1309] daphne s2/s5 corrupted and offline; Toolserver; Blocker Bug <10https://jira.toolserver.org/browse/TS-1309> (Marlen Caemmerer) [23:10:28] 3(assigned) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (Marlen Caemmerer) [23:10:33] 3(work started) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (Marlen Caemmerer) [23:11:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [23:12:20] 3(commented) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (Marlen Caemmerer) [23:15:01] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [23:15:01] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:16:01] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [23:18:01] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [23:19:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:21:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:01] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [23:30:02] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [23:54:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42379 MB (4% inode=99%): [23:54:15] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [23:54:15] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:54:15] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [23:54:15] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:54:15] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [23:55:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default