[00:07:02] sql is down, you already know it, right? [00:11:08] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:15:02] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [00:15:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:16:00] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [00:18:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [00:19:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:22:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:25:00] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [00:31:01] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [00:51:09] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [00:51:09] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:52:40] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [00:52:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:52:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [00:55:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:01:19] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1896.000000 [01:04:52] [[User:Merlissimo/Sandbox]] N 10https://wiki.toolserver.org/w/index.php?oldid=6689&rcid=8836 * Merlissimo * (+12337) (https://wiki.toolserver.org/w/index.php?title=Job_scheduling&oldid=6030) [01:09:22] 3(created) [SWMTBOT-45] Make SWMTBot recongnize commands based on its current nickname instead of its set nickname in SWMTBot.ini; SWMTBot; Minor Improvement <10https://jira.toolserver.org/browse/SWMTBOT-45> (Seahorseruler ) [01:11:09] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [01:15:08] Alchimista: sql is available, "only" sql-s2-user/sql-s2-rr is down [01:15:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:16:00] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [01:17:01] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [01:17:21] 3(created) [TS-1310] Add "seahorseruler" to CVN multi-maintainer group; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1310> (Krinkle) [01:18:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [01:19:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:22:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:25:01] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [01:31:09] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [01:41:40] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [01:51:11] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:51:11] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [01:52:41] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:52:41] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [01:52:41] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [01:53:47] [[Admin:Next maintenance]] 10https://wiki.toolserver.org/w/index.php?diff=6690&oldid=6631&rcid=8837 * Pathoschild * (+98) (/* User wishes */ + [[jira:1308]] (Install XML_Serializer PEAR package), fixed typos) [01:54:58] [[Admin:Next maintenance]] M 10https://wiki.toolserver.org/w/index.php?diff=6691&oldid=6690&rcid=8838 * Pathoschild * (+3) (/* User wishes */ fix) [01:55:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:11:19] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:15:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:16:00] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [02:17:01] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [02:18:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [02:19:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:23:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:25:59] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [02:31:09] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [02:41:40] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6271.000000 [02:51:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:51:19] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [02:52:39] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [02:52:39] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:52:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [02:56:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:11:20] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:15:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:16:01] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [03:17:59] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [03:18:20] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [03:19:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:23:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:25:59] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [03:31:09] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [03:41:40] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7335.000000 [03:51:41] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [03:52:19] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:52:40] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [03:52:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:52:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [03:57:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:10:50] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3318.000000 [04:12:18] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:15:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:16:09] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [04:16:49] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1575.000000 [04:18:10] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [04:19:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:19:20] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [04:24:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:25:49] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1930.000000 [04:26:10] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [04:31:09] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [04:52:27] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:52:39] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [04:52:49] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [04:52:49] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:52:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [04:57:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:12:26] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:16:09] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [05:16:17] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:19:08] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [05:19:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:19:19] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [05:23:27] There seems to be an automatic HTTPS => HTTP redirect when a directory URL has no ending slash. For example: https://toolserver.org/~pathoschild/magicredirect (redirects to HTTP) versus https://toolserver.org/~pathoschild/magicredirect/ (no redirect). [05:23:36] Is this a known / upstream issue? [05:24:12] You're sure it's a HTTPS issue and not bad rewrite rules on your end? [05:24:19] an HTTPS * [05:25:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:25:36] I guess your rewrite rules don't try to catch trailing /. [05:25:49] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 8384 MB (0% inode=99%): [05:25:49] I'd file a ticket in the TS' JIRA. [05:26:13] Also, forty lashes for not sanitizing output. [05:26:19] Where do you want them? [05:27:09] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [05:31:18] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [05:31:57] [05:32:25] You gotta escape in there too. [05:43:13] Joan: Should be fixed. [05:43:39] (Some ancient code in a formatFormValue() method.) [05:44:08] As for the redirect, I already checked my rewrite.script, and it seems to happen with other users' tools too. [05:45:54] Yeah, probably not a Zeus bug, probably something on the TS' end. [05:46:05] Either way, you want a JIRA ticket for that. [05:46:21] Not sure if anyone is handling that kind of thing right now. Maybe nosy or DaB. Dunno. [05:46:27] https://jira.toolserver.org/browse/TS-1228 [05:46:55] Hey, look at that. [05:46:59] Also https://jira.toolserver.org/browse/TS-1130 [05:47:02] Someone probably configured a helpful redirect on the TS. [05:47:12] And it proabbly just needs a tweak. [05:47:14] That's my guess. [05:48:19] If you can catch DaB. on IRC and help him find the cause, it should be easy to fix. [05:48:24] To get fixed, I mean. [05:50:41] * Pathoschild nods. [05:52:24] 3(resolved) [TS-1228] Redirection from Https address are done using Http <10https://jira.toolserver.org/browse/TS-1228> (Jesse Plamondon-Willard) [05:52:27] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:52:37] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [05:53:12] Pathoschild: You're due for many lashes, young man. [05:53:18] Pfft. [05:53:48] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [05:53:48] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:53:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [05:53:52] Many of these scripts were originally written when I barely knew PHP. [05:54:09] * Pathoschild remembers the innocent young days, when I didn't even know what SQL injection was. [05:54:43] Laziness is a constant battle with knowledge. [05:54:45] Such is life. [05:54:56] is in * [05:55:01] I really just give up on keyboards. [05:55:07] I've had enough of them. [05:57:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:02:48] Load avg. on willow is WARNING: WARNING - load average: 21.12, 14.81, 11.60 [06:07:48] Load avg. on willow is OK: OK - load average: 11.14, 14.90, 12.75 [06:12:26] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:16:18] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:17:08] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [06:20:08] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [06:20:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:20:18] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [06:26:05] Joan: Looks like you're not the only one testing for proper output escaping: http://en.wikipedia.org/wiki/Cross-site_scripting?diff=477687011&oldid=476677613 :) [06:26:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:27:09] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [06:32:18] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [06:52:26] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:52:48] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [06:53:49] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [06:53:49] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:53:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [06:57:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:12:27] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:16:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:17:19] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [07:20:18] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [07:20:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:20:18] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [07:26:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:27:17] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [07:32:19] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [07:39:29] [[Special:Log/newusers]] create 10 * Ali0987 * (New user account) [07:52:27] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:53:48] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [07:54:49] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [07:54:49] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:54:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [07:57:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:02:57] Load avg. on willow is WARNING: WARNING - load average: 13.95, 15.33, 12.78 [08:03:57] Load avg. on willow is OK: OK - load average: 11.23, 14.34, 12.59 [08:12:27] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:16:28] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:17:18] ethernet 0/1/11 [zedler] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/11:DOWN: 1 int NOK : CRITICAL [08:20:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:20:28] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [08:21:17] ethernet 0/1/19 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/19:UP: 1 int NOK : CRITICAL [08:26:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:27:18] ethernet 0/1/21 on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/21:UP: 1 int NOK : CRITICAL [08:31:04] [[User talk:Ali0987]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6692&rcid=8840 * Ali0987 * (+56) (Created page with "--~~~~") [08:32:28] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [08:36:56] [[Special:Log/move]] move 10 * Ali0987 * (moved [[02User talk:Ali098710]] to [[UT:Ali0987]]: This does not suit) [08:50:48] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:52:28] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:53:49] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [08:54:49] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [08:54:49] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:54:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [08:57:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:01:20] 3(commented) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (Marlen Caemmerer) [09:13:27] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [09:18:45] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:20:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:20:54] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [09:21:13] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1952.000000 [09:26:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:32:54] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [09:49:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3632.000000 [10:13:03] Load avg. on willow is WARNING: WARNING - load average: 17.26, 14.21, 12.08 [10:14:03] Load avg. on willow is OK: OK - load average: 13.82, 13.88, 12.10 [10:15:03] Marcin Cieslak * Re: [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [10:16:14] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:16:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:16:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:16:33] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [10:16:33] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:16:34] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:16:34] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [10:16:44] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [10:19:44] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:20:53] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [10:21:02] Marcin Cieslak * Re: [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [10:21:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:26:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:32:54] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [10:47:02] Marlen Caemmerer * Re: [Toolserver-l] Unplanned maintenance at daphne /s2+s5-user [10:49:14] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7232.000000 [10:54:54] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3559 [10:55:14] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3067.000000 [10:56:54] MySQL slave on z-dat-s4-a is OK: Uptime: 1282353 Threads: 5 Questions: 45461121 Slow queries: 56617 Opens: 9803 Flush tables: 1 Open tables: 519 Queries per second avg: 35.451 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1722 [10:57:14] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1407.000000 [11:16:14] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:16:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:16:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:16:43] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [11:17:34] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [11:17:34] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [11:17:34] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:17:34] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [11:21:03] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [11:21:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:26:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:27:23] 3(commented) [TS-1309] daphne s2/s5 corrupted and offline <10https://jira.toolserver.org/browse/TS-1309> (Marlen Caemmerer) [11:33:04] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [11:51:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42124 MB (4% inode=99%): [12:16:14] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:16:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:16:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:16:44] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [12:17:34] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [12:17:34] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [12:17:34] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:17:34] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [12:21:03] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [12:21:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:26:11] hello all [12:26:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:32:30] Hello DaBPunkt [12:32:42] hallo nosy :) [12:33:03] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [12:36:57] nosy: commons is importing on cassia AFAIU, and daphne's status is? [12:37:21] DaBPunkt: daphne has awfully corrupted innodb files [12:37:29] i cannot even dump the *wiki [12:37:37] asked asher for a dump of s2 [12:38:00] i moved the ibdata and ib log files to / to get some space [12:38:12] but repair table does not work either [12:38:29] ok [12:38:31] when i try to dump the server goes offline when i am at an error point of the dump [12:39:25] tried to connect daphne to the san and i see daphnes controller on the san but when i map a volume it does not show on daphne [12:39:44] Do you have time to finish the SAN-thing on daphne? [12:40:00] that should be prio 1 [12:40:02] the san thing [12:40:21] sebaso said i should look if i have trouble on rosemary or adenia too [12:40:30] ts-prio1, yes :-) [12:40:44] so get the overall status first and then troubleshoot further [12:40:48] :) [12:40:59] my personal prio 1 for ts yes [12:41:23] we also try to reach the data center regarding nightshade [12:41:30] did you dump commons from z-dat-s4-a to a file? [12:41:50] DaBPunkt: you also think nightshade simply not sees its hard disks? [12:41:52] yes i did [12:42:03] before i stopped the slave and recorded the position [12:42:03] ok, we will need the dump for daphne later [12:42:29] it took only 2 hours to make the dump [12:43:03] nosy: I am unsure about nightshade. Maybe it is just the solaris. I will start a debian-install and look if the installer sees the discs or now later today [12:43:03] "only to hours" - i become humble... [12:43:16] good idea [12:43:22] -or [12:43:26] i was not able to start any installation at all [12:43:45] I trained with yarrow on monday [12:43:46] pxe would not work although i enabled dhcp [12:43:57] you got it to boot? [12:44:20] yes, I sent you an email too [12:44:24] if yarrow became nightshade at first that would be good too i suppose [12:44:26] oh [12:44:45] much work on puppet is to do [12:44:52] I already started that too [12:45:05] great [12:48:15] nosy: Did you or the geschäftstelle made fotos from the old SAS-card of yarrow before you sent it back? [12:48:29] or noticed the exact type? [12:49:02] Because now we have a simple SAS-card in yarrow and I thought that we have had a RAID-card before [12:50:01] (the difference is a aditional "i" or "r" on the type AFAIU the internet) [12:50:40] hang on [12:50:42] i look [12:55:54] DaBPunkt: i dont have a photo but can you remember we exchanged it against one in willow or nightshade which had a bbu and ours did not? i can only find that i read on it ucs-51 and the dell people said there was an i controller built in [12:56:42] DaBPunkt: found the foto [12:56:44] http://www.monoro.de/nosy/P1000342.JPG [12:57:55] backside is http://www.monoro.de/nosy/P1000344.JPG [12:58:53] mm, no type afais [13:08:22] but afair the new one looked really the same [13:16:14] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:16:19] nosy: it doesn't realy matter [13:16:44] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [13:17:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:17:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:18:33] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [13:18:33] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [13:18:33] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:18:33] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [13:21:03] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [13:21:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:21:44] nosy: The debian-installers sees the disc. I will finish the installation and look if it is booting [13:26:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:33:03] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [13:39:39] DaBPunkt: debian installer on nightshade? [13:39:43] nosy: booted successfully :-) [13:39:46] yes [13:39:48] great [13:40:00] then solaris is not what we are used to? [13:40:30] nosy: I never installed solaris on the TS, that was always rivers job. I have no idea who he does it [13:40:38] does → did [13:41:05] DaBPunkt: i wandered if mark inserted a boot cd at first [13:41:28] nosy: most of our servers have no cdrom-drive [13:41:43] DaBPunkt: but dont we get in conflict since sge is not running on linux? [13:42:22] DaBPunkt: how about usb cdrom? [13:43:17] nosy: there is SGE for debian from source AFAIR [13:43:33] DaBPunkt: that would be great [13:43:46] I very doubt that Merlissimo has a solaris-cluster at home too ;) [13:44:14] and my university use SGE too on a cluster and I doubt that there are using sge too [13:45:31] DaBPunkt: on debian you won't have any resource control features [13:46:19] so it starts jobs until the server crashs undr the load? [13:46:27] Merlissimo: which we dont really use anyway ;) [13:47:06] sge uses it partly. [13:47:17] but it will take severall days anyway until I have puppet ready for debian [13:47:22] ah ok [13:47:32] DaBPunkt: no, but one user can use all resources [13:48:12] ok, that's a minor problem I think [13:48:19] (and to current sge version 6.2u3 only support kernels until 2.6.1) [13:48:55] so then you have to update it ;-) [13:50:05] i still have red hat 5.3 cds [13:50:47] I still have debian woody-cds somewhere… [13:50:52] wed have a proper cron and updates and apache out of the box [13:51:02] and probably even mono [13:51:05] and all 64 bit [13:51:32] and free solaris-licenses for more db-servers [13:52:11] but like I said: Ruhig Blut, ich brauche mehrere Tage um zum. die Grundinstallationen mit puppet hinzubekommen. Kein Stress [13:52:11] but no sun studio [13:52:58] nosy: At the beginning, I userland-server would be stay on solaris nevertheless until everyone switched to debian [13:53:05] nosy: At the beginning, 1 userland-server would be stay on solaris nevertheless until everyone switched to debian [13:53:39] DaBPunkt: ok [13:54:09] Merlissimo: Is there a way to tell SGE to set the arch to "sol-amd64" by default? [13:54:17] for jobs I mean [13:54:25] yes [13:54:32] ok [14:00:55] Sun Grid Engine execd on yarrow is CRITICAL: Connection refused by host [14:00:55] Environment on yarrow is CRITICAL: Connection refused by host [14:01:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:01:04] Load avg. on yarrow is CRITICAL: Connection refused by host [14:01:23] MySQL slave on yarrow is CRITICAL: Cant connect to MySQL server on yarrow (146) [14:01:23] / on yarrow is CRITICAL: Connection refused by host [14:01:23] NTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:34] SMF on yarrow is CRITICAL: Connection refused by host [14:01:34] /aux0 on yarrow is CRITICAL: Connection refused by host [14:01:43] MySQL on yarrow is CRITICAL: Cant connect to MySQL server on yarrow (146) [14:01:44] s2 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [14:01:44] s4 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [14:01:44] s5 replag on yarrow is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on yarrow (146) [14:01:45] SMTP on yarrow is CRITICAL: Connection refused [14:01:45] /tmp on yarrow is CRITICAL: Connection refused by host [14:01:45] Cluster on yarrow is CRITICAL: Connection refused by host [14:03:05] nosy: I shut-downed and de-powered nightshade again to not confuse people who try to login [14:04:22] DaBPunkt: fine, no problem [14:04:35] my focus is on the san topic then anyway [14:05:26] yes, bringing daphne back online is urgend :) [14:16:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:17:01] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [14:17:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:17:29] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:18:39] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [14:18:39] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [14:19:02] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [14:19:02] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:21:09] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [14:22:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:27:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:33:09] SMF on daphne is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:default [15:16:20] s2 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:17:10] FC 0/7 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/7:UP: 1 int NOK : CRITICAL [15:17:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:17:29] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:18:39] MySQL on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:18:40] s5 replag on daphne is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on daphne (146) [15:19:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:19:59] MySQL slave on daphne is CRITICAL: Cant connect to MySQL server on daphne (146) [15:21:20] FC 0/6 on fsw1-n1-oe16-esams.mgmt is CRITICAL: FC port 0/6:UP: 1 int NOK : CRITICAL [15:22:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:25:22] 3(created) [MNT-1203] Re-Install yarrow; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1203> (DaB.) [15:27:10]