[00:11:10] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1929.000000 [00:11:40] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1947 [00:43:10] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71215 MB (6% inode=99%): [00:44:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70725 MB (5% inode=99%): [00:48:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [00:51:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:52:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 328459 MB (6% inode=34%): [01:11:09] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3389.000000 [01:12:39] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3460 [01:15:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [01:15:39] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3607 [01:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [01:51:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:52:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 327480 MB (6% inode=34%): [02:06:00] /sql on thyme is WARNING: DISK WARNING - free space: /sql 187872 MB (19% inode=99%): [02:15:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4317.000000 [02:15:39] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4323 [02:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:50:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [02:51:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:52:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325589 MB (6% inode=34%): [02:55:38] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3595 [02:56:30] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3568.000000 [03:12:10] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71267 MB (6% inode=99%): [03:13:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70657 MB (5% inode=99%): [03:40:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71041 MB (6% inode=99%): [03:41:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70234 MB (5% inode=99%): [03:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:50:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [03:52:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:52:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325636 MB (6% inode=34%): [03:55:39] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2965 [03:56:30] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2948.000000 [04:31:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 72039 MB (6% inode=99%): [04:32:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70599 MB (5% inode=99%): [04:34:29] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1800.000000 [04:34:39] MySQL slave on thyme is OK: Uptime: 462825 Threads: 17 Questions: 196044544 Slow queries: 66389 Opens: 37894 Flush tables: 1 Open tables: 2976 Queries per second avg: 423.582 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1787 [04:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:50:10] /sql on cassia is WARNING: DISK WARNING - free space: /sql 72002 MB (6% inode=99%): [04:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [04:51:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70001 MB (5% inode=99%): [04:52:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:52:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325492 MB (6% inode=34%): [05:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [05:53:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:53:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325418 MB (6% inode=34%): [06:48:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [06:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:54:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:54:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325321 MB (6% inode=34%): [07:20:19] Nothing in my time-to-render logs suggests that the toolserver is slow. BUT people some programmers aren't using indexes and instead relying and testing with query cache that speeds things up. [07:22:28] What I have been getting is "Database Error: Lost connection to MySQL server at 'reading initial communication packet', system error: 0 on sql-s4/commonswiki_p" when I am trying to run http://tools.wikimedia.de/~daniel/WikiSense/CommonsClash.php [07:25:59] I've been getting that alot more recently. No clue where it comes from, although a misbehaving bot today leads to think too many new connections. [07:31:19] tested -- nope [07:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:50:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [07:54:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:54:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325236 MB (6% inode=34%): [07:57:11] [[Special:Log/newusers]] create 10 * Iphonegratis * (New user account) [08:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [08:52:54] [[Special:Log/newusers]] create 10 * WendyOds * (New user account) [08:55:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:55:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325110 MB (6% inode=34%): [09:33:44] [[Special:Log/newusers]] create 10 * Rolandhelper * (New user account) [09:48:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [09:55:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:55:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324931 MB (6% inode=34%): [10:18:46] 3(commented) [TS-1434] Think about integration of graphserv <10https://jira.toolserver.org/browse/TS-1434> (Marlen Caemmerer) [10:18:47] 3(created) [TS-1434] Think about integration of graphserv; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1434> (Marlen Caemmerer) [10:34:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 74067 MB (6% inode=99%): [10:35:09] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 69952 MB (5% inode=99%): [10:41:49] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36722 MB (9% inode=99%): [10:42:02] DaB. * [Toolserver-l] Update of SGE on Thursday 5. July [10:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:50:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:50:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [10:52:50] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23556 MB (5% inode=99%): [10:55:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:55:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324840 MB (6% inode=34%): [11:07:15] hello [11:10:49] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35258 MB (8% inode=99%): [11:16:49] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 57720 MB (14% inode=99%): [11:17:48] 3(commented) [TS-1434] Think about integration of graphserv <10https://jira.toolserver.org/browse/TS-1434> (DaB.) [11:22:34] hello all [11:28:45] 3(commented) [TS-1434] Think about integration of graphserv <10https://jira.toolserver.org/browse/TS-1434> (Marlen Caemmerer) [11:30:54] Hello [11:31:04] will start the maintenance now [11:31:33] today: solaris patch day, reboots afterwards if necessary, and updates of fisheye and jira [11:35:42] 3(commented) [TS-1434] Think about integration of graphserv <10https://jira.toolserver.org/browse/TS-1434> (DaB.) [11:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:51:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:51:50] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [11:55:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:55:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324757 MB (6% inode=34%): [11:55:53] DaBPunkt, could you confirm/deny that this guy is allowed to run this bot on TS as long as it's not "continuous"? https://en.wikipedia.org/wiki/Wikipedia:Bots/Requests_for_approval/MahdiBot_3 [11:58:12] MaxSem: I will leave a message there later today [11:59:44] thanks! [12:09:19] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:37] DaBPunkt: some on dewiki http://de.wikipedia.org/wiki/Wikipedia:Bots/Antr%C3%A4ge_auf_Botflag#2012-05-30_.E2.80.93_MahdiBot [12:14:00] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:45:00] jira.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.477 second response time [12:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:51:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:51:49] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [12:53:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:55:19] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.035049 secs [12:56:00] SMF on web.amaranth is OK: OK - all services online [12:56:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:56:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324691 MB (6% inode=34%): [12:56:59] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 169 bytes in 0.472 second response time [12:57:59] jira.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.470 second response time [13:04:20] NTP on yarrow is OK: NTP OK: Offset 0.091473 secs [13:14:27] [[Special:Log/newusers]] create 10 * Shriram * (New user account) [13:36:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:41:00] SMF on web.amaranth is OK: OK - all services online [13:48:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:48:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:52:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:52:00] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [13:56:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:56:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324606 MB (6% inode=34%): [14:02:04] DaB. * Re: [Toolserver-announce] [Toolserver-l] Maintenance at Thursday, 28.06.2012 [14:07:01] /sql on thyme is WARNING: DISK WARNING - free space: /sql 185465 MB (19% inode=99%): [14:31:49] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:00] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40136 MB (9% inode=99%): [14:43:29] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:48:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:49:39] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2176.000000 [14:49:39] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2183 [14:52:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:52:01] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [14:53:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:53:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 156 bytes in 0.471 second response time [14:54:00] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63508 MB (15% inode=99%): [14:54:01] jira.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.471 second response time [14:56:00] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 400 Bad Request - 156 bytes in 0.477 second response time [14:56:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:56:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324505 MB (6% inode=34%): [14:58:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:05:20] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:02] SMTP on willow is OK: SMTP OK - 0.004 sec. response time [15:19:07] wow...jira is brandnew now... [15:19:20] works for me - can you please check if you see errors? [15:25:13] the updates finished in the maintenance window...since we announced we reboot tomorrow i just would like to test turnera [15:31:09] nosy as I said last night: Nothing in my time-to-render logs suggests that the toolserver is slow. BUT people some programmers aren't using indexes and instead relying and testing with query cache that speeds things up. [15:32:20] Dispenser: true. thats one of the problems - you never know - is it the system or the code? i really would like to say: "this reference code is working - must be your problem then" [15:32:33] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:19] but i also got some debug code from a user that tells me the times to answer a certain request from the userdb-server adenia are varying extremly [15:38:43] I've got nothing that uses that host, so sorry [15:42:25] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [15:49:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:49:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:50:29] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3552 [15:51:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3631 [15:53:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:53:33] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [15:54:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:57:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:57:47] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324366 MB (6% inode=34%): [15:58:30] jira.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.479 second response time [15:59:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:00:22] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 241 bytes in 0.538 second response time [16:06:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:53] jira.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 170 bytes in 0.478 second response time [16:18:54] SMF on web.amaranth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:21:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:25:37] Load avg. on amaranth is CRITICAL: CRITICAL - load average: 11.74, 59.79, 37.05 [16:32:47] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:39] Load avg. on amaranth is WARNING: WARNING - load average: 0.68, 9.12, 19.67 [16:40:40] Load avg. on amaranth is OK: OK - load average: 0.53, 3.76, 14.30 [16:42:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4789.000000 [16:49:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:50:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:52:04] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5316 [16:52:47] Hi [16:53:02] I' using cat-scan to find wrongly tagged Non-free images [16:53:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:09] but it seems to be taking a while [16:53:46] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [16:57:36] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:58:28] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324247 MB (6% inode=34%): [17:00:19] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:06:30] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:33:03] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6025.000000 [17:49:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:50:36] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:52:27] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6263 [17:53:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:54:09] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [17:58:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:58:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324162 MB (6% inode=34%): [18:00:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:06:34] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:31:57] what's wrong wit mysql? [18:33:36] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:26] ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [18:43:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6671.000000 [18:43:28] DaBPunkt: ^^ i can't connect anywhere now :-/ [18:46:58] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317887 MB (5% inode=34%): [18:50:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:51:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:53:02] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6789 [18:53:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:54:24] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [18:55:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321652 MB (6% inode=34%): [18:58:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:00:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:06:38] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:34:10] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:11] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6258.000000 [19:50:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:51:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:53:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5926 [19:54:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:55:06] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321578 MB (6% inode=34%): [19:55:47] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [19:59:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:01:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:06:40] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:34:12] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4838.000000 [20:50:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:51:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:53:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4505 [20:54:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:55:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321456 MB (6% inode=34%): [20:56:39] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [21:00:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:02:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:07:31] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:26:02] [[Special:Log/newusers]] create 10 * Killbedbugs7 * (New user account) [21:34:14] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2163.000000 [21:35:06] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:47] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3595 [21:39:28] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3584.000000 [21:46:09] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2163 [21:51:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:52:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:55:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:56:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321375 MB (6% inode=34%): [21:57:34] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [21:58:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [22:01:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:03:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:08:27] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:09] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3606 [22:12:10] MySQL slave on rosemary is OK: Uptime: 6677655 Threads: 16 Questions: 2681635302 Slow queries: 491684 Opens: 12573 Flush tables: 6 Open tables: 661 Queries per second avg: 401.583 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1764 [22:12:41] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1762.000000 [22:24:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:35:33] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:24] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 67944 MB (5% inode=99%): [22:52:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:52:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:55:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:57:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321290 MB (6% inode=34%): [22:58:12] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [22:59:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4392.000000 [23:01:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:03:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:08:46] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:38] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4255 [23:24:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:35:51] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:53:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:55:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:57:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319941 MB (6% inode=34%): [23:58:27] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL